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Abstract 


This report looks at student growth in Arkansas school districts disaggregated by district 
poverty and by the district’s value-added performance relative to other districts. We estimated 
district value-added performance statistics by subject and grade level (8 and 11-12) for 
longitudinal student cohorts, using statistical models that adjusted for district demographics, 
the percentage of students in the analysis, and prior student scores. We found that differences 
in these value-added statistics across districts at each poverty level were large enough to be of 
practical importance to educators and policymakers. 


In addition to value-added statistics, we also calculated unadjusted descriptive statistics 
related to changes in student achievement over time—such the percentage of students at 

a given achievement level in an earlier year who reached the On-Track level in the current 
year—for districts classified as above or below average based on the value-added statistics. 
At all poverty levels, we found substantial differences in these statistics between districts 
identified as above and below average in the models. However, even in above-average 
districts, the great majority of Off-Track students were not reaching On-Track benchmarks for 
college and career readiness in English, mathematics, reading, and science. No group of high- 
poverty districts was able to catch up at least a third of their Off-Track students or more than 
11% of their Far-Off-Track students in any subject. This is reason to pay increased attention 
to promising but sometimes underemphasized long-term strategies for improving student 
outcomes. 


viii 


Comparisons of Student Growth by District 
Performance and Poverty’ 


Introduction 


A substantial body of literature has focused on the role of school districts in supporting 
improvement in student learning (Hightower, Knapp, Marsh, & McLaughlin, 2002; Marsh et al., 
2005; Supovitz, 2006; Maclver & Farley-Ripple, 2008; ACT, 2012a; Daly & Finnigan, 2016). 
As such, school districts have been a significant focus of education reform efforts (Whitehurst, 
Chingos, & Gallaher, 2013; Chingos, Whitehurst, & Gallaher, 2013) and public recognition 
(Broad Foundation, 2014). 


School districts are potentially important because they are often the administrative unit closest 
to students that can oversee improvement strategies spanning preschool through grade 12 
(ACT, 2012a). For example, working closely with teachers and school leaders, district leaders 
can work to ensure students receive a content-rich curriculum in each subject aligned across 
elementary, middle, and high school levels; establish assessment and data systems to monitor 
student progress and follow students as they change schools; promote educators’ use of those 
systems (Dougherty, 2015a, 2015b); ensure that time is set aside in every school for teachers 
to collaborate; develop a coaching system for teachers; and lead efforts to involve parents and 
community leaders. District leaders can benefit from systematic, ongoing feedback on how well 
these practices are implemented in their district (Dougherty, 2016). 


In theory, we would expect differences in educator practices across districts to account for 

a substantial share of cross-district differences in student outcome indicators, such as test 
scores, high school graduation rates, and college enrollment rates. In practice, the relationship 
between differences in practices and student outcomes across districts has been difficult to 
investigate. Information is usually missing on what practices are being implemented, and 

how well, in which districts. Second, practices are not generally introduced separately in 
randomized trials, so that it is difficult to demonstrate a causal relationship between changes in 
specific practices and changes in student performance. Third, practices may be implemented 
inconsistently within districts, so that a given practice may be found only in some schools and 
classrooms. This dilutes the overall impact of the practice on district-wide outcomes.” 


In connection with this issue, it is worth investigating whether differences in outcomes across 
districts serving similar student populations are large enough to inspire us to study differences 
in their practices. Does there appear to be a large enough “district effect” to make it worthwhile 
to investigate the cause? This question was explored by Whitehurst, Chingos, and Gallaher 
(2013), who studied differences in student performance in fourth- and fifth-grade mathematics 
across districts in Florida and North Carolina after adjusting for differences in student 
demographics. They found that a one-standard-deviation difference in district performance 
statistics amounted to a difference in test scores similar to the gains associated with seven 

to 12 weeks of instruction. This is of practical importance, given that performance in the 5th- 
and 95th-percentile districts might be expected to differ by about 3.3 standard deviations, or 


' This study uses data maintained by the Arkansas Department of Education and is published with its permission. 


? Variation in practices across schools and classrooms within districts can be related to variation in student outcomes at 
the school and classroom levels, with the usual need to disentangle alternative causal explanations for these student 
outcome differences. 
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between 23 and 39 weeks of instruction. The latter estimate amounts to more than a standard 
school year of 36 weeks. 


We explored cross-district performance differences in two previous reports. In the first report 
(Dougherty & Shaw, 2016a), we found that differences in model-generated performance 
statistics between above- and below-average districts were large enough to be of practical 
importance. However, the districts identified as above- and below-average varied by grade 
and subject, so that a district with better-than-average performance in elementary school 
mathematics might not be above average in high school mathematics or in elementary school 
reading.? The second report (Dougherty & Shaw, 2016b) classified districts into three poverty 
categories—high-poverty, medium-poverty, and lower-poverty districts—and looked at the gap 
in model-generated district performance statistics between above- and below-average districts 
in each poverty category. Our disaggregation of districts by poverty category was motivated by 
the idea that efforts to improve teaching and learning are likely to be particularly important for 
economically disadvantaged students, who are more likely to start out far behind academically 
and have more trouble catching up when they are behind (Stanovich, 1986; ACT, 2012b; 
Dougherty, 2014). 


In addition to comparing districts on performance statistics that adjust for student 
demographics, the second report looked at gaps between districts on unadjusted descriptive 
student achievement statistics, such as average student scores and the percentages of 
students who were On Track or Far Off Track.* These gaps were also of practical importance 
based on the larger number of additional students who would be On Track and the smaller 
number who would be Far Off Track if student achievement in below-average districts could 
replicate that in above-average districts. Despite these promising results, the report also found 
that among medium- and high-poverty districts, no group of districts got the majority of their 
students to meet On-Track benchmarks in reading, mathematics, or science. 


The current report follows up on the first two by comparing districts using statistics related 

to student growth—change in achievement over time. As in the second report, we classified 
districts into three poverty levels—high-poverty, medium-poverty, and lower-poverty—and 
compared the model-generated performance statistics of above- and below-average districts 
at each poverty level. In the current report, district performance statistics were calculated using 
value-added models that took prior student scores into account. In addition, we compared 
districts on unadjusted descriptive student growth statistics—percentages of students at 
different achievement levels in a prior year reaching the On-Track level in the current year, and 
change in average scores. 


Specifically, this report addresses the following questions: 


1. What were the differences in district value-added performance statistics between above- 
and below-average districts at each poverty level, and were they large enough to be of 
practical importance? 

2. Did value-added performance vary more in one poverty category than in another? For 


example, did we see a larger variation in value-added performance statistics among high- 
poverty districts? 


3 District performance also varied based on the statistical model used to calculate the performance statistics. Each of the 
models controlled for student demographics, but the models differed on whether they also controlled for district average 
student demographics, prior student test scores, and district average prior student scores. 


2 4 See the Data section of this report for definitions of “On Track,” “Off Track,” and “Far Off Track.” 


3. How much did the change in average student scores relative to On-Track targets differ 
between above- and below-average districts at each poverty level? Were these differences 
large enough to be of practical importance? 


4. How much did the percent of previously On-Track students who stayed On Track differ 
between above- and below-average districts at each poverty level? Were these differences 
large enough to be of practical importance? 


5. How much did the percent of previously Off-Track and Far-Off-Track students who got On 
Track differ between above- and below-average districts at each poverty level? Were these 
differences large enough to be of practical importance? 


6. In what grades and subjects did specified percentages of previously On-Track students 
remain On Track in districts at different poverty and performance levels? 


7. In what grades and subjects did specified percentages of previously Off-Track and Far-Off- 
Track students get On Track in districts at different poverty and performance levels? 


In this report, we use the phrase “district value-added performance statistics” to refer to 
model-generated estimates of differences in student scores between districts that control for 
student and district demographics and prior student achievement levels. These statistics were 
used to answer Questions 1 and 2. We use “student growth statistics” to refer to unadjusted 
descriptive statistics on changes in student achievement over time. These statistics were used 
to answer Questions 3 through 7. In general, unadjusted student growth statistics such as 
changes in average scores and movement of students across achievement levels are better 
at addressing the question, “How well did the students do?” District value-added performance 
statistics are better at addressing the question, “How well did the district do?” In all cases, we 
classified districts as above-average, average, or below-average based on their value-added 
performance statistics, not their unadjusted student growth statistics.® 


Data 

This report used longitudinal cohorts created from student-level enrollment and test data 
supplied by the Arkansas Department of Education for the 2006-07 through the 2013-14 
school years. The test data we used included student scores on the Arkansas Benchmark 
Exams (ABE) in grade 4, the ACT Explore® tests in grade 8, and the ACT® in grades 11 and 12. 
All enrollment and test datasets contained state-encrypted student IDs so that records for the 
same students could be linked anonymously across enrollment and test datasets. 


Our statistical analysis used data both on students’ individual demographic characteristics 

and prior test scores and the demographics and average prior test scores of their districts. 
This required three steps to create the necessary datasets: (1) create longitudinal cohorts 

of students to be included in the analysis, coding individual students into demographic and 
academic achievement level categories; (2) calculate district-level statistics and apply rules for 
including districts in the analysis; and (3) merge student- and district-level datasets together 
based on the district in which each student was enrolled. 


5 Cross-district comparisons may look different depending on whether value-added performance statistics or unadjusted 
student achievement statistics are used. For example, a district with above-average value-added performance but more 
disadvantaged students may have a lower level of student growth than a district with average value-added statistics but 
more advantaged students. 
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1. Creation of Student Cohorts and Calculation of Student-Level 
Statistics 


This section describes how longitudinal cohorts of students were created for the analysis, how 
their demographic characteristics were derived from the Arkansas state enrollment and test 
data, and how students were classified into achievement levels (On Track, Off Track, and Far 
Off Track) based on their test results. 


Creation of student cohorts. For the analysis using eighth-grade test scores, we began with 
students enrolled in fourth grade in the initial cohort year (for example, the 2006-07 school 
year) and followed them forward for four subsequent school years, keeping students who 
were enrolled in the district the entire time and who took both fourth-grade ABE tests in the 
initial cohort year and all four ACT Explore tests in eighth grade in the final cohort year. For 
the analysis using eleventh- and twelfth-grade scores, we followed initial-year eighth-grade 
students forward for four subsequent years, keeping students who were twelfth graders four 
years later who had been enrolled in the district the entire time, and who took all four 

ACT Explore tests in the initial cohort year and all four ACT tests in twelfth grade in the final 
cohort year or in eleventh grade in the next to final year. We used the most recent score for 
students who took the ACT more than once in grades 11 and 12. 


This process created four longitudinal cohorts at each level, referred to as the 2007-11, 
2008-12, 2009-13, and 2010-14 cohorts based on the initial cohort year and the final cohort 
year four years later.®° Thus, there were eight total student cohorts (four each for grades 4—8 
and 8-12). At each grade level (4—8 and 8-12), we concatenated the four student-level cohorts 
into a single dataset in order to avoid double-counting students who were retained in the 

initial cohort grade and to create an indicator variable for those retained students. In all, the 
73,633 students in our four 4—8 cohorts and the 36,377 students in our four 8—12 longitudinal 
cohorts comprised respectively 51% and 25% of the students enrolled in the initial cohort years 
(Appendix A, Tables A1—A3).” 


We separated out students who had been enrolled in the same district for multiple years—as 
opposed to using snapshot data on all students enrolled in the final year—in order to focus on 
those students whose test results would be more likely to reflect the instructional program in 
the district where they were tested. We followed students forward from an initial cohort year— 
rather than starting with tested students in the final year and looking backward—in order to 
account for student attrition between grades 8 and 12, when many students in some districts 
may drop out.® We required that students in the 4-8 and 8-12 cohorts have test scores from 
the initial year because the value-added statistical models we used controlled for those prior 
test scores. 


§ In this nomenclature, school years are named after their spring semesters, so that students in a 2007-11 cohort were 
present in the district from the collection of enrollment data in the fall of the 2006-07 school year to the collection of test 
data in the spring of the 2010-11 school year. 


N 


As clarified in Table A3, most of the difference in percentages of students in the analysis between grades 8-12 and 4-8 
is due to differences in the percentage of students who were enrolled in the final grade four years later (accounting for 
26 vs. 16 percentage points attrition from the sample in 8—12 vs. 4-8, likely due to high school dropout) and differences 
in the percentage of students taking the test in the initial grade (accounting for 28 vs. 4 percentage points attrition in 
8-12 vs. 4-8, because many students in the grades 8-12 cohorts did not take ACT Explore in eighth grade, but nearly 
all students in the 4-8 cohorts took the ABE in fourth grade). The state began paying for students to take ACT Explore 
in the 2010-11 school year, which benefitted all of the grades 4-8 cohorts in their final cohort years but none of the 
grades 8-12 cohorts in their initial cohort years. 


ow 


A backward- or forward-looking cohort selection process can create the same student cohort, with the only difference 
being the denominator to which the size of the cohort is compared. 


Identification of students’ demographic and program participation status. Students’ 
characteristics may vary naturally over time: For example, a student’s family may qualify for the 
free- and reduced-price lunch program when the student is in fourth grade but not when the 
same student is in eighth grade. Likewise, a student’s special education and English language 
learner (ELL) status or a student's self-identified ethnicity may change over time. Because an 
indicator of low-income, English language learner, or special education status may signal a 
level of disadvantage even if the status is not consistent every year, we identified students as 
low-income, English language learners, or special education if they had that status in either 
the initial or final cohort grade level, e.g., either fourth or eighth grade for the students in the 
grades 4—8 analysis. Because no such logic applies to inconsistent reporting of student ethnic 
status, we used the student’s reported ethnicity in the earliest cohort year as the determining 
factor for the student's overall ethnic status.? Students whose ethnicity, low-income status, 
special education status, or English language learner status could not be ascertained using 
these criteria were dropped from the analysis. Less than 1% of records were dropped based 
on incomplete demographic data (table notes in Appendix A, Tables A1 and A2)."° 


Classification of students by academic achievement level. In grades 8 and 11-12, we 
classified students as On Track in a given subject if they met or exceeded the ACT College 
Readiness Benchmark on ACT Explore (in grade 8) or the ACT (in grades 11-12) in the subject 
in question."' To identify prior student achievement levels in fourth grade, we used the ABE 
On-Track targets calculated for a previous study (Dougherty, Hiserote, & Shaw, 2014), which 
identified the fourth grade ABE score in literacy and mathematics associated with a 50% or 
better probability of meeting or exceeding the eighth-grade Benchmark on ACT Explore in the 
corresponding subject.’? In turn, Off-Track students were defined as those missing the On- 
Track Level by one standard deviation or less in the grade and subject in question, while Far- 
Off-Track students scored more than a full standard deviation below the On-Track Level. These 
criteria resulted in the definitions for On-Track, Off-Track, and Far-Off-Track achievement 
levels shown in Table 1. 


° The only exception was for a student with missing ethnic data for the earliest cohort grade (e.g., fourth grade in the 
grades 4—8 analysis) but ethnic data present for the final cohort grade level (e.g., eighth grade), in which case we used 
the data from the final grade level. 


‘© The percentage of records dropped due to incomplete data was around 0.1% in grades 4-8 and 8-12. 

" The ACT College Readiness Benchmarks, updated in 2013, identify the ACT scores associated with a 50% probability 
of earning a B or approximately a 75% chance of earning a C in entry-level college courses corresponding to the ACT 
subject tested (Allen & Sconing, 2005; Allen, 2013). In turn, the ACT Explore Benchmarks identify the scores on that test 
associated with a 50% probability of reaching the Benchmark in the corresponding subject on the ACT (Allen, 2013). 

‘2 The analysis linked student-level fourth grade ABE scores in 2007 and 2008 with the same student’s ACT Explore 
scores in the 2010-11 and 2011-12 school years. The eighth-grade ACT Explore reading test was treated as the closest 
same-subject match to the fourth-grade ABE literacy test, which covers both reading and writing. 
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Table 1. Scale Score Ranges for On Track, Off Track, and Far Off Track"? 


On Track Off Track Far Off Track 
Grade 4 Arkansas 
Benchmark Exam 
Literacy 772 and above 586-771 585 or below 
Mathematics 675 and above 575-674 574 or below 
Grade 8 ACT Explore 
English 13-25 9-12 8 or below 
Mathematics 17-25 14-16 13 or below 
Reading 16-25 12-15 11 or below 
Science 18-25 15-17 14 or below 
Grade 11/12 ACT 
English 18-36 12-17 11 or below 
Mathematics 22-36 17-21 16 or below 
Reading 22-36 16-21 15 or below 
Science 23-36 18-22 17 or below 


Calculation of changes in students’ scores relative to the On-Track level. We calculated 
each student's score relative to the On-Track level in the initial and final grades, measured 

in standard deviation units, and defined the difference between these two relative scores 

as the change in the student’s score relative to the On-Track level. For example, consider a 
hypothetical student with an ACT Explore reading score of 13 and an ACT reading score of 20. 
Measured in score points, this student scored three points below the On-Track level of 16 on 
the ACT Explore reading test and two points below the On-Track level of 22 in ACT reading. 
Measured in standard deviation units, the student scored 3/3.9 = 0.77 standard deviations 
below the On-Track level on ACT Explore and 2/6.3 = 0.32 standard deviations below the 
On-Track level on the ACT, for a gain of (-0.32) — (-0.77) = 0.45 standard deviation units.‘ We 
used these changes in standardized scores relative to the On-Track level as our measure of 
student growth or progress, as it did not depend on having a common scale across the two 
tests. No common scale existed in grades 4—8 between the ABE and ACT Explore tests. 


2. District-Level Statistics and Inclusion of Districts in the Analysis 

Our calculation of district-level statistics began with 238 K-12 school districts that were in 
existence continuously from the 2006-07 through the 2013-14 school years.'® Since our focus 
was on traditional K-12 school districts, charter schools that were not part of such a district 
were omitted from the analysis.'® For these 238 districts, we calculated statistics on district- 


‘3 Standard deviations on the ACT Explore tests were 4.2 points in English, 3.5 in mathematics, 3.9 in reading, and 3.3 
in science (ACT, 2013, Table 4.11). Standard deviations on the ACT were 6.5 in English, 5.3 in mathematics, 6.3 in 
reading, and 5.3 in science (ACT, 2014, Table 5.4). Standard deviations on the grade 4 ABE, calculated for all students 
tested in the 2006-07 and 2007-08 school years, were 186.37 scale score points in literacy and 100.93 in mathematics. 


14 A student who was the same number of score points below the On-Track level on the ACT Explore and the ACT would 
be shown as making progress, as standard deviations on the ACT are larger than those on ACT Explore. 


'8 If District A consolidated into District B at any time between the 2006-07 and 2013-14 school years, then A’s students 
were combined with B’s for the years prior to the consolidation and everyone was treated as part of District B. Thus, 
basing the analysis on the 238 districts that existed after consolidation did not, in itself, reduce the number of students in 
the analysis. 

1® Omitting students in charter schools that were not part of a K-12 district reduced the number of students in the 
analysis—after the other rules for inclusion were applied—by 199 students in grades 4-8 and 86 students in 
grades 8-12. 


wide demographics and the district's number and percentage of students included in the 
analysis. Next, we classified districts as rural or non-rural, identified districts that were eligible 
for the analysis, and divided the eligible districts into the three poverty categories used in this 
report. 


District-wide demographics. We used each district’s fall student-level enrollment data for 
kindergarten through twelfth grade for each year from 2006-07 through 2009-10 to derive 
annual statistics on the overall district-wide percentage of low-income, African American, 
Hispanic, Asian, White, Native American, English language learner, and special education 
students. These statistics were used as district-level predictors in our statistical models."” 


District percentage of students in the analysis. For each cohort, we calculated the number 
of students in the analysis in each district as a percentage of the total number of students 
enrolled in the district in the initial cohort grade and year. If a district's percentage of students 
in the analysis is low compared with other districts—reflecting a higher rate of student mobility 
or a lower percentage of students taking the test—this might either raise or lower the relative 
performance of the district. For example, if students whose families face the most challenges 
leave the sample in disproportionate numbers, that could bias the results in favor of districts 
with high attrition.’® High attrition could also result from the presence of a nearby military base 
or from the district’s being less effective at retaining and educating students. In the last of 
these cases, controlling for attrition rates in the statistical model picks up some of the district 
performance we are trying to measure. Further research may explore the variables that 

are associated with student attrition to identify when attrition should be treated as a district 
performance indicator (e.g., as in the case of high school dropout rates) and when it is simply 
an aspect of the environment in which the district operates. 


Rural district status. Using school-level information from the 2012-13 Common Core of Data 
(U.S. Department of Education, 2014), we defined as “rural” any district in which all schools 
have a two-digit NCES locale code beginning with 3 (small town) or 4 (rural). 


Selection of districts for inclusion in the analysis. We applied two additional criteria to 
identify which of the 238 continuously existing regular K-12 districts should be included in the 
analysis: 


Accuracy of low-income statistics. The use of students’ low-income status as an important 
control in the statistical models made the accuracy of this classification an important 
consideration. To assess the accuracy of each district’s low-income statistics in a given year, 
we regressed the district’s overall percentage of low-income students in that year on 

U.S. Census estimates of poverty rates of individuals age 5—17 in the district to get a statewide 
relationship between the two variables, which in turn yielded a Census-predicted district low- 
income percentage for each year.'® To have its students included in the analysis, a district’s 


‘7 The district-wide demographic statistics calculated this way differ from ones that would be calculated by aggregating our 
cohort data, which do not cover all grades. As was the case in our student cohorts, we dropped students with missing 
demographic data when calculating the district-wide statistics. 


‘8 The poverty measure based on students’ free- and reduced-price lunch status is an imperfect measure of those 
challenges, so using this measure as a predictor in the statistical models does not completely adjust for this possible 
bias. 


8 Census-defined poverty uses a lower income threshold than the state definition of low income, which is based on 
federal eligibility requirements for the free- and reduced-price school lunch program. Thus we needed to derive a 
predicted low-income percentage from the Census data rather than just using the Census percentage. We hypothesized 
that a district with accurate low-income data would have a relationship between the two poverty measures that is not too 
different from the state average relationship between the two measures. 
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percentage of low-income students in kindergarten through grade 12 had to fall within 

20 percentage points of its Census-predicted value in each school year from 2006-07 through 
2009-10, the starting years for the cohorts in this report. Of 238 Arkansas K—12 districts, 

202 met this requirement. 


Number of students in the analysis. To be included in the analysis for a given grade level 

(4-8 or 8-12), districts were required to have at least 20 students in the four combined 
longitudinal cohorts for that grade level. In grades 4-8, all 202 districts that met the income 
data requirement also met this criterion, despite the fact that Arkansas has many small districts 
(Appendix B, Figure B1 and Table B1). Because of low ACT Explore and ACT participation 
rates, 33 of the 202 districts meeting the low-income data criterion had fewer than 20 students 
eligible for the grades 8-12 analysis, leaving 169 eligible districts at that grade level.?' 


Disaggregation of districts into poverty categories. We calculated district-wide 
percentages of low-income students across the initial cohort years, 2006-07 through 2009-10, 
and used these cross-year percentages to classify school districts in the study into three 
poverty categories: 


* Lower poverty: >20—50% low-income students 
* Medium poverty: >50%—70% low-income students 
* High poverty: >70% low-income students 


We selected these categories because the bottom and top categories each accounted for just 
under one-quarter of the 202 Arkansas districts included in the grades 4-8 analysis, while just 
over half of the eligible districts were contained in the middle category (Appendix B, Figure 
B2). Arkansas had no districts with 20% or fewer low-income students. 


3. Combining Student- and District-Level Data 

At each grade level (4—8 and 8-12), we merged the concatenated file containing student-level 
data on the four cohorts with the district-level data created in the previous step, based on the 
district in which each student was enrolled. This process created a single dataset at each level 
with matched student- and district-level data. 


Methods 


Once the datasets for the study were built, our analysis had four steps: (1) Use statistical 
models to estimate district value-added performance statistics for each subject and grade 
level; (2) use these value-added statistics to classify districts into above-average, average, or 
below-average performance categories by subject and grade level; (3) calculate descriptive 
district-level student growth statistics; (4) aggregate the model-based district value-added 
performance statistics and the descriptive student growth statistics to address the research 
questions in the study. We describe these steps here. 


20 We refer to the analysis of eighth-grade test scores that takes fourth-grade test scores into account as a grades 4-8 
analysis, and the analysis of test scores in grades 11 and 12 that took eighth-grade test scores into account as a 
grades 8-12 analysis. 

1 The differences between the third and fourth data columns in Appendix A, Tables A1—A2, show the effect that removing 
ineligible districts (and charter schools) had on the number of students in the analysis. The percentages in the last 
column are based on the number of students in eligible districts. 


1. Statistical Models Used to Create District Value-Added Performance 
Statistics 
We used similar sets of student- and district-level predictors to predict student-level scores on 
each of the four eighth-grade ACT Explore tests and four ACT tests for students in 
grades 11-12 (Tables 2 and 3).?2 The models (one per subject and grade level) contained 
student-level predictors measuring students’ low-income, ethnic, English language learner, 
special education status, and prior test scores; and district-level averages of these predictors. 
The district-level averages might be related to a school district's academic culture, funding, and 
priorities; these influences might in turn affect students’ test scores. The models also contained 
predictors on the district's number and percentage of students included in the analysis, in order 
to explore effects of district size and cohort attrition, respectively. We also included a dummy 
variable for whether the district was located in a rural area, on the theory that that might affect 
teacher recruitment and thus, indirectly, student performance. In addition, dummy variables for 
three of the four student cohorts were included to allow for shifts in average test scores across 
years. 


We refer to the models in Tables 2 and 3 as value-added models because they included 
students’ prior achievement in the set of variables used to predict students’ current 
achievement. The statistics generated by the models may be thought of as answering 

the question: How did students in this district perform relative to what would have been 
predicted for students with the same demographics and prior scores in districts with the 

same demographics and average prior scores? These statistics differed from most of those 
produced by state accountability reports on student status in three major ways. First, our 
statistical models used student scores, not proficiency status, as the dependent variable. 
Second, accountability reports often do not adjust for students’ and districts’ demographic 
characteristics, as policymakers do not want to convey the message that educators are free 

to aim for lower test results for disadvantaged students. Third, many state accountability 
reports do not adjust for students’ prior scores (student growth reports being the exception), 
emphasizing student achievement levels in the current year without regard to the same 
students’ performance in prior years. As such, model-based statistics might be hypothesized to 
do a better job of reflecting relative district effectiveness, and model-based statistics that take 
prior scores into account might do a better job of reflecting relative district effectiveness over a 
particular grade span. The regression coefficients associated with student characteristics allow 
for differences in the degree of difficulty in educating students, just as degree of difficulty is 
taken into account in scoring Olympic gymnastics or diving events. 


22 These value-added models are similar to Model 4 in Dougherty & Shaw (2016a), except that Model 4 did not include a 
variable for districts’ rural status. 
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Table 2. Predictors in the Models for Grades 4-8 
Type of Data Predictor 


Student-Level Intercept 
Low-income status 
African American status 
Hispanic status 
Asian status 
Native American status 
English Language Learner status 
Special education status 
Flag for retained student* 
Ath-grade literacy score 
4th-grade mathematics score 
Number of years between tests 
Flag for cohort ending in 2012 
Flag for cohort ending in 2013 
Flag for cohort ending in 2014 


District-Level % low-income students 
% African American students 
% Hispanic students 
% Asian students 
% Native American students 
% English Language Learner students 
% special education students 
District average 4th-grade literacy score 
District average 4th-grade mathematics score 
Number of students in model 
% of students in model 


Flag for rural district 


* Retained students included in the analysis had enrollment records in consecutive initial cohort years and met the other 
inclusion criteria. 
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Table 3. Predictors in the Models for Grades 8-12 
Type of Data Predictor 


Student-Level Intercept 
Low-income status 
African American status 
Hispanic status 
Asian status 
Native American status 
English Language Learner status 
Special education status 
Flag for retained student* 
8th-grade English score 
8th-grade mathematics score 
8th-grade reading score 
8th-grade science score 
Number of years between tests 
Flag for cohort ending in 2012 
Flag for cohort ending in 2013 
Flag for cohort ending in 2014 


District-Level % low-income students 
% African American students 
% Hispanic students 
% Asian students 
% Native American students 
% English Language Learner students 
% special education students 
District average 8th-grade English score 
District average 8th-grade mathematics score 
District average 8th-grade reading score 
District average 8th-grade science score 
Number of students in model 
% of students in model 


Flag for rural district 


* Retained students included in the analysis had enrollment records in consecutive initial cohort years and met the other 
inclusion criteria. 


Because the models used in the study contained both student- and district-level predictors, 
we estimated them as hierarchical linear models.”° We used the district-level random effect 
estimated by the model as the district performance statistic for the grade and subject in 
question. This process generated four sets of district performance statistics at each grade 
level, one for each ACT Explore or ACT subject. 


2. Classification of Districts into Relative Performance Categories 


For each subject and grade level, we defined as “above-average” those districts whose value- 
added performance statistics fell in the top quintile for the grade and subject in question and 


23 SAS Proc Mixed was used for all of the statistical models in this report. Information on the SAS code used for the 
models is available on request. Fixed effect coefficients estimated by the models are shown in Appendix G. 
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also were statistically different from average at the .05 confidence level. Similarly, “below- 
average” districts were those in the bottom quintile whose performance statistics were different 
from average at the .05 confidence level. Districts not meeting these requirements—i.e., in the 
middle three quintiles, or in the top or bottom quintile but not statistically different from average 
at the .05 level—were classified as average.” 


Although each district is in the same poverty category throughout, the same district might fall 
into different performance categories in different subjects and grade levels. For example, a 
district could be above average in grades 4-8 mathematics and below average in grades 8-12 
reading.*° Thus, a performance category such as “above-average districts” comprised different 
districts depending on the grade and subject. 


3. District-Level Student Growth Statistics 


In the next step, we created datasets containing the following district-level student growth 
statistics for each subject area across grades 4—8 and 8-12: 


* the district average change in student scores relative to the On-Track level; 


* the district's percentage of On-Track students in the earlier grade who stayed On Track 
in the later grade; 


* the district's percentage of Off-Track students in the earlier grade who reached the On- 
Track benchmark in the later grade; 


* the district's percentage of Far-Off-Track students in the earlier grade who reached the 
On-Track benchmark in the later grade; 


* the district average change in student scores relative to the On-Track level for students 
who were On Track, Off Track, or Far Off Track in the earlier grade (one statistic per 
district per prior achievement level). 


District-level growth statistics were calculated from student-level datasets that contained data 
on students from all four cohorts; the statistics were aggregated across the four cohorts, rather 
than being calculated for each cohort separately. For eighth-grade reading, we used fourth- 
grade ABE literacy as the prior test for identifying which students were On Track, Off Track, or 
Far Off Track leaving fourth grade. 


4. Aggregation of Achievement Statistics by District Performance 

and Poverty 
For each grade and subject, we calculated weighted averages of the district value-added 
performance statistics and student growth statistics across districts in each poverty and 
performance category, using the number of students in the analysis in each district as the 
weights. We used the average district performance statistics to address the first question in 
the study, and the student growth statistics to address the third through the seventh question. 
To address the second research question—whether district effect variances differ by poverty 
category—we calculated the unweighted average variance of the district performance statistics 
within each poverty category. We did pairwise statistical tests (F-tests) of whether these 


4 Top- or bottom-quintile districts were more likely to be classified as average if they had smaller numbers of students in 
the analysis. 


25 We explored the frequency with which this sort of inconsistency occurs in Dougherty & Shaw (2016a). 
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variances differed at a .05 significance level between pairs of poverty categories in a given 
subject—for example, whether the variances were different in high- and medium-poverty 
districts in eighth-grade mathematics. 


Limitations 

Though this report looked at district performance statistics, we were not able to differentiate 
“district effects” from “school effects.” Thus, we did not make a distinction between “value- 
added performance of the district in grades X to Y” and “value-added performance of the 
district's school(s) in grades X to Y.” The majority of Arkansas school districts are small 

and rural, and many districts have only one school at a given level. For example, in 2014, 
156 (92%) of the 169 districts in the grades 8—12 cohort analysis had only one high school 
serving grades 11 and 12. Likewise, 180 (89%) of the 202 districts in the grades 4—8 cohort 
analysis had only one school serving eighth grade. Thus, for the great majority of Arkansas 
districts, the performance statistic in grades 8-12 could also be thought of as an indicator of 
the performance of the district’s single high school. The comparable statistic for eighth grade 
could be used as an indicator of the performance between grades 4 and 8 of the district’s 
single middle or junior high school and its feeder elementary school(s).”° The value of treating 
the district as the unit of analysis is to focus attention on the district's potential to improve its 
schools systematically across the elementary, middle, and high school levels. 


Second, we did not attempt to compare the wide range of statistical models that could be used 
to generate district value-added performance statistics. Our goal was to examine results from 
relatively straightforward value-added models that control for generally available student- and 
district-level demographic statistics. We did not refine the models to eliminate variables that did 
not add much explanatory power to the models. 


Third, we studied measured district performance differences with the understanding that these 
differences may reflect the effects both of educator practices and of unmeasured student, 
parent, and community influences that were not picked up as controls in the statistical analysis 
(Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010). For example, some districts may 
operate in more favorable community environments than other districts with similar student 
demographics. Thus, the measured performance differences such as those discussed in this 
report should be treated as the starting point for further inquiry into why these differences exist 
and what can be done to improve student outcomes in all school systems. 


Fourth, our focus on specific goals in Questions 6 and 7 was for illustrative purposes. 
Educators and policymakers may choose to set different goals. 


Fifth, because the data in this report are for Arkansas students and districts, further research in 
other states is needed to determine how the results generalize across states. One goal of this 
study is to encourage this research. 


Finally, because of concerns about the statistical reliability of results from small student groups, 
we did not report results for groups of fewer than 20 students. 


76 In a state with a number of larger districts, one could partition the variance in performance across schools in those 
districts into the variance across districts and the variance across schools within districts. 
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Results 


Question 1: Were differences in district value-added performance 
statistics between above- and below-average districts at each poverty 
level large enough to be of practical importance? 

Tables 4 and 5 compare the district value-added statistics of above- and below-average 
districts by district poverty level, as estimated using the value-added models described in 
Tables 2 and 3. For example, in eighth-grade English, students in the seven above-average 
lower-poverty districts performed on average 0.11 standard deviations above predicted based 
on the variables in the model (Table 3).?” Likewise, students in the eight below-average lower- 
poverty districts scored an average of 0.12 standard deviations below predicted. Thus, the 
performance gap between the two groups of districts was 0.23 standard deviations.” 


In general, the performance differences between above- and below-average districts estimated 
using a value-added model, as shown in Tables 4 and 5, were smaller than those estimated 
using a status model (Dougherty & Shaw, 2016b, Tables 4 and 5).”° Nonetheless, these 
differences were still large enough to be of practical significance. For example, the differences 
in eighth-grade scores shown in Table 4 may be compared with average growth per year 
between grades 4 and 8 on the ABE exam ranging from 0.24 to 0.30 standard deviations 

in literacy and from 0.27 to 0.35 standard deviations in mathematics (depending on student 
cohort), calculated for the three study cohorts in Dougherty and Shaw (2016a). Differences 

in eleventh or twelfth grade scores in Table 5 may be compared with average growth per 

year between the ACT Explore and ACT exams of 0.26 standard deviations in English, 0.28 

in mathematics, 0.31 in reading, and 0.24 in science (ACT, 2012c), using the average of the 
ACT Explore and ACT standard deviations to convert typical growth in score points to standard 
deviations. Thus, the difference in statistically adjusted student performance between above- 
and below-average districts in the same poverty category frequently came close to a year’s 
typical student growth. 


27 The size of a single standard deviation in score points is shown by subject and test in footnote 13. 

28 Appendix F provides information on district-wide demographics by poverty and performance group. For example, the 
demographics of the districts compared in eighth-grade English are shown in Appendix F, Table F1. 

22 If districts with stronger student growth between fourth and eighth grades also have higher fourth-grade scores, the 
district random effect in an eighth-grade status model would reflect the impact of those higher fourth-grade scores. By 
contrast, an eighth-grade value-added model would adjust out the effect of the higher fourth-grade scores. 
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Table 4. Value-Added Performance of Above- and Below-Average Districts 


in Grades 4-8, by District Poverty 


Lower Poverty Medium Poverty High Poverty 
Average Average Average 
District of District of District of District 
Performance # of Performance # of Performance # of Performance 
Subject Level Districts Statistics Districts Statistics Districts Statistics 
English Above average 7 0.11 14 0.12 4 0.10 
Below average 8 -0.12 17 -0.10 5 -0.11 
Difference 0.23 0.22 0.21 
Mathematics Above average 15 0.16 16 0.16 0.16 
Below average -0.16 21 -0.15 -0.14 
Difference 0.32 0.31 0.30 
Reading Above average 6 0.15 13 0.13 0.11 
Below average -0.12 10 -0.13 6 -0.14 
Difference 0.26 0.26 0.26 
Science Above average 11 0.13 12 0.14 0.17 
Below average 7 -0.19 18 -0.14 -0.17 
Difference 0.32 0.28 0.35 


Apparent discrepancies in the differences shown in the tables in this report are due to rounding; the differences shown are 
between the unrounded numbers. 


Table 5. Value-Added Performance of Above- and Below-Average Districts 
in Grades 8-12, by District Poverty 


Lower Poverty Medium Poverty High Poverty 
Average Average Average 
District of District of District of District 
Performance # of Performance # of Performance # of Performance 
Subject Level Districts Statistics Districts Statistics Districts Statistics 
English Above average 8 0.09 7 0.12 3 0.11 
Below average 3 -0.13 8 -0.10 4 -0.10 
Difference 0.22 0.22 0.22 
Mathematics Above average 9 0.17 12 0.12 0.14 
Below average -0.11 12 -0.14 -0.14 
Difference 0.27 0.25 0.28 
Reading Above average 6 0.11 0.08 2 0.10 
Below average -0.12 -0.08 -0.11 
Difference 0.23 0.17 0.21 
Science Above average 5 0.10 0.07 1 0.11 
Below average -0.12 -0.10 1 -0.11 
Difference 0.22 0.17 0.23 
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Question 2: Did value-added performance vary more in one poverty 
category than in another? 

Theory does not provide us with a clear expectation of whether value-added performance 
differences should be larger among high- or lower-poverty districts. We might hypothesize that 
in lower-poverty districts, a larger share of student learning takes place outside of school— 
which is why average student achievement is generally higher in those districts. In that case, 
in-school factors might account for a smaller share of learning differences across districts. On 
the other hand, we do not know how much out-of-school learning varies across districts or 
whether that variation is greater in lower-poverty districts. Nor do we know whether educator 
practices vary more across high-poverty than across lower-poverty districts or how these 
differences in practices are likely to affect variations in district performance. 


Tables 4 and 5 do not appear to show a consistent pattern in the size of value-added 
performance differences across districts in the three poverty-level categories. For example, 
looking from left to right in Table 4, lower-poverty districts had the largest performance gaps 

in grades 4—8 English (0.23 of a standard deviation, versus 0.22 and 0.21 for medium- and 
high-poverty districts) and mathematics (0.32 of a standard deviation, versus 0.31 and 0.30 for 
medium- and high-poverty districts). In science, the gaps were largest for high-poverty districts 
in science (0.35 of a standard deviation, versus 0.32 and 0.28 for lower- and medium-poverty 
districts). As these cases illustrate, many of these differences were small. Some comparisons 
were based on small numbers of districts: For example, the difference between above- and 
below-average high-poverty districts in science in grades 8-12 was based on a single district 
in each performance category (Table 5). 


To explore this issue in a way that examines variation across all districts, including those 
not statistically different from average, we calculated the unweighted standard deviation and 
variance (the square of the standard deviation) of the value-added performance statistics 
for districts in each poverty category by grade and subject, and conducted a set of statistical 
significance tests comparing these variances between district poverty categories. Table 6 
shows the standard deviations of the district value-added statistics and the results of the 
statistical tests. 


Table 6. Comparison of Within-Category Performance Variation between 
Poverty Categories 


Statistical Significance 


Performance Variation of Comparison 
within Poverty Category between Categories 
Lower- Medium- High- Lower 
Poverty Poverty Poverty Lower vs. Medium 
Grade Span Subject Districts Districts Districts vs. High Medium vs. High 
4-8 English 0.083 0.072 0.067 
Mathematics 0.125 0.101 0.088 “ * 
Reading 0.086 0.080 0.085 
Science 0.109 0.094 0.108 
8-12 English 0.070 0.062 0.063 
Mathematics 0.103 0.082 0.088 * 
Reading 0.066 0.049 0.056 id 
Science 0.060 0.043 0.047 +e 


***Significant at the .01 level. **Significant at the .05 level. “Significant at the .10 level. 
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From Table 6, we can see that the variation in performance was at least slightly larger for 
lower-poverty districts in all cases, but in only a few cases was the difference between poverty 
categories large enough to be statistically significant. For example, there were no statistically 
significant differences in the variation in performance between medium- and high-poverty 
districts (last column of Table 6). In twelfth-grade reading and science, the difference between 
lower- and medium-poverty districts was significant at the .05 level, but the (smaller) difference 
between lower- and high-poverty districts was not. Thus, we are unable to conclude from this 
evidence that the variation of performance across districts varies systematically by district 
poverty level. 


Question 3: How much did the change in student scores relative to On- 
Track targets differ between above- and below-average districts at each 
poverty level? Were these differences large enough to be of practical 
importance? 

In grades 4—8, students in above-average districts made substantially more academic progress 
than did their counterparts in below-average districts, as measured by the change in student 
scores relative to the On-Track target (Table 7, shown in more detail in Appendix C, 

Table C1). For example, in high-poverty districts, eighth-grade students in above-average 
districts were 0.10 of a standard deviation closer to the On-Track target in eighth-grade 
mathematics than they were as fourth graders relative to the fourth-grade target. In below- 
average districts, however, eighth graders were 0.10 of a standard deviation farther below 
the eighth-grade target than they were as fourth graders relative to the fourth-grade target. 
This amounts to a difference in student progress between above- and below-average districts 
of 0.20 of a standard deviation (Table 7). Overall, these differences between above- and 
below-average districts ranged from 0.20 of a standard deviation for high-poverty districts in 
mathematics to 0.37 of a standard deviation among medium-poverty districts in reading. As 
with the differences in random effects in Table 4, the differences in Table 7 may be compared 
with average growth per year in ABE in grades 4—8 ranging from 0.24 to 0.30 standard 
deviations in literacy and 0.27 to 0.35 standard deviations in mathematics. By this measure, 
even the smallest difference between above- and below-average districts amounted to more 
than half of a year’s student progress.°° 


In grades 8-12, differences in student progress between above- and below-average districts 
ranged from 0.15 of a standard deviation for medium-poverty districts in science to 0.93 of a 
standard deviation among high-poverty districts in science (Table 8). The largest difference 
with more than one district on both sides of the comparison was 0.37 of a standard deviation 
for high-poverty districts in mathematics, which may be compared with average mathematics 
growth per year between the ACT Explore and the ACT of 0.28 of a standard deviation (ACT, 
2012c).°*' Thus, these differences ranged from more than half a year to well over a year of 
typical student growth. 


3° The progress measure in grades 4—8 reading is based on the ABE literacy test in fourth grade and the ACT Explore 
reading test in eighth grade, although the literacy test also assesses writing. No progress measures were available for 
English and science. However, value-added measures were available for all four ACT Explore subjects, based on the 
statistical models in Table 2 that use fourth-grade literacy and mathematics scores as predictors of ACT Explore scores 
in each subject. 

31 Appendix C, Table C1 expands on the information in Table 7, and Tables C2 and C3 expand on the information in 
Table 8, including information on average districts, numbers of students, and average scores in the prior and current 
grades. 
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Table 7. Average Changes in Scores Relative to On-Track Targets in Grades 4—8 


Lower Poverty Medium Poverty High Poverty 
District Number Change Number Change Number Change 
Performance of in of in of in 
Subject Level Districts Scores Districts Scores Districts Scores 
Mathematics Above average 15 0.09 16 0.10 3 0.10 
Below average 9 -0.16 21 -0.21 3 -0.10 
Difference 0.25 0.31 0.20 
Reading Above average 6 0.34 13 0.34 0.28 
Below average 0.09 10 -0.03 0.05 
Difference 0.25 0.37 0.22 


Table 8. Average Changes in Scores Relative to On-Track Targets in Grades 8-12 


Lower Poverty Medium Poverty High Poverty 
District Number Change Number Change Number Change 
Performance of in of in of in 
Subject Level Districts Scores Districts Scores Districts Scores 
English Above average 8 -0.04 7 -0.05 3 -0.03 
Below average 3 -0.25 8 -0.26 4 -0.30 
Difference 0.21 0.21 0.27 
Mathematics Above average 9 0.06 12 -0.01 0.12 
Below average -0.16 12 -0.21 4 -0.25 
Difference 0.22 0.20 0.37 
Reading Above average 6 0.21 0.20 2 0.30 
Below average -0.04 4 -0.05 1 -0.16 
Difference 0.25 0.25 0.46 
Science Above average 5 -0.01 -0.01 1 0.56 
Below average -0.24 2 -0.16 1 -0.37 
Difference 0.23 0.15 0.93 


Question 4: How much did the percent of previously On-Track students 
who stayed On Track differ between above- and below-average districts 
at each poverty level? Were these differences large enough to be of 


practical importance? 


Differences between above- and below-average districts in the percentage of fourth-grade On- 


Track students staying On Track in eighth grade ranged from 11 percentage points for high- 


poverty districts in mathematics to 28 percentage points for high-poverty districts in reading 


(Table 9).°2 In grades 8-12, these differences ranged from five percentage points for medium- 


32 Apparent discrepancies between the percentages in Tables 9-14 and their differences are due to rounding; the 
differences are calculated using the unrounded percentages. Also, the numbers of students in Tables 9-14 cannot be 
added up across subjects without double-counting students who would change their status in more than one subject. 


Table 9. Percentages of On-Track Fourth-Grade Students Staying On Track 
in Grade 8, by District Poverty and Performance 


Simulated 
Number of Additional 
On-Track On-Track 
Number Number Grade 4 Grade 8 
of % of % Difference Students Students 
District Above- Staying Below- Staying in On- in Below- in Below- 
Poverty Average On Average On Track Average Average 
Subject Category Districts Track Districts Track Rates Districts Districts 
Mathematics Lower 15 19% 9 60% 19% 1,484 289 
Medium 16 76% 21 56% 20% 1,672 329 
High 3 70% 59% 11% 87 10 
Reading Lower 77% 64% 13% 1,208 160 
Medium 13 14% 10 59% 15% 826 125 
High 3 12% 6 43% 28% 265 75 


poverty districts in English to 35 percentage points for high-poverty districts in mathematics, 


where less than a third of On-Track eighth graders were still On Track in grade 12 in the four 


below-average districts (Table 10). 


To provide context for how important these differences are, we can simulate how many 


additional previously On-Track students in the below-average districts would have stayed On 


Track had those districts matched the percentage in the above-average districts. Results of 


these simulations are shown in the rightmost column of Tables 9 and 10. For example, looking 


at lower-poverty districts in mathematics (Table 9), increasing the percentage of students 


staying On Track in eighth grade in the nine below-average lower-poverty districts from 


60% to 79% to match the percentage in the 15 above-average lower-poverty districts would 


have resulted in 289 additional students remaining On Track. By comparison, increasing the 


percentage of students staying On Track in mathematics in the three below-average high- 


poverty districts by 11 percentage points to match the corresponding percentage in the three 


above-average districts would add only 10 On-Track eighth-grade students, because the 


number of On-Track fourth-grade students in the below-average districts—87 students—was 


relatively low to begin with.*? 


33 In all, there were 87 On-Track, 151 Off-Track, and 146 Far-Off-Track fourth graders in the three below-average high- 
poverty districts in grades 4-8 mathematics, as shown in Tables 9, 11, and 13, respectively. 
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Table 10. Percentages of On-Track Eighth-Grade Students Staying On Track 
by Grade 12, by District Poverty and Performance 
Simulated 


Number of Additional 
On-Track On-Track 


Number Number Grade 8 Grade 12 
of % of % Difference Students Students 
District Above- Staying Below- Staying in On- in Below- _ in Below- 
Poverty Average On Average On Track Average Average 
Subject Category Districts Track Districts Track Rates Districts Districts 
English Lower 8 91% 3 79% 12% 486 56 
Medium 7 85% 8 80% 5% 1,668 77 
High 3 74% 4 62% 12% 264 31 
Mathematics Lower 9 79% 7 62% 17% 761 132 
Medium 12 71% 12 50% 21% 495 103 
High 5 67% 4 32% 35% 143 50 
Reading Lower 6 83% 4 63% 20% 310 63 
Medium 5 78% 4 68% 10% 313 31 
High 2 65% 1 NR NR 16 NR 
Science Lower 5 73% 3 52% 20% 221 44 
Medium 3 69% 2 51% 18% 51 9 
High 1 NR 1 NR NR 14 NR 


NR = Not Reported because the calculation is based on a group of fewer than 20 students. 


Question 5: How much did the percent of previously Off-Track and Far- 
Off-Track students who got On Track differ between above- and below- 
average districts at each poverty level? Were these differences large 
enough to be of practical importance? 

Off-Track Students. Differences between above- and below-average districts in the 
percentage of fourth-grade Off-Track students getting On Track in eighth grade ranged from six 
percentage points for high-poverty districts in mathematics to 17 percentage points for lower- 
and medium-poverty districts in the same subject (Table 11). In grades 8-12, these differences 
ranged from 10 percentage points for medium-poverty districts in English and mathematics 
and high-poverty districts in science to 21 percentage points for medium-poverty districts in 
reading (Table 12). 


For context, we can simulate how many additional Off-Track students in the earlier grade in 
the below-average districts would have been On Track in the later grade had those districts 
matched the percentage getting On Track in the above-average districts (Tables 11 and 

12). For example, increasing the percentage of students getting On Track in eighth-grade 
mathematics in the nine below-average lower-poverty districts from 21% to 38% would result 
in 284 additional Off-Track students getting On Track (Table 11). On the other hand, increasing 
the percentage of students getting On Track by six percentage points in the three below- 
average high-poverty districts would add only nine students getting On Track on those districts 
(Table 11). 


Table 11. Percentages of Off-Track Fourth-Grade Students Getting On Track 
in Grade 8, by District Poverty and Performance 


Simulated 
Number of Additional 
On-Track On-Track 
Number Number Grade 4 Grade 8 
of % of % Difference Students Students 
District Above- Getting Below- Getting in On- in Below- in Below- 
Poverty Average On Average On Track Average Average 
Subject Category Districts Track Districts Track Rates Districts Districts 
Mathematics Lower 15 38% 9 21% 17% 1,675 284 
Medium 16 35% 21 18% 17% 1,906 332 
High 3 20% 14% 6% 151 9 
Reading Lower 6 38% 28% 9% 1,384 131 
Medium 13 35% 10 20% 15% 1,108 166 
High 3 25% 13% 11% 716 82 
Table 12. Percentages of Off-Track Eighth-Grade Students Getting On Track 
by Grade 12, by District Poverty and Performance 
Simulated 
Number of Additional 
Off-Track On-Track 
Number Number Grade 8 Grade 12 
of % of % Difference Students Students 
District Above- Getting Below- Getting in On- in Below- _ in Below- 
Poverty Average On Average On Track Average Average 
Subject Category Districts Track Districts Track Rates Districts Districts 
English Lower 8 41% 3 23% 18% 193 34 
Medium 7 35% 8 25% 10% 597 59 
High 3 27% 4 9% 18% 261 47 
Mathematics Lower 9 31% 7 19% 12% 628 76 
Medium 12 22% 12 11% 10% 561 59 
High 5 16% 4 4% 12% 225 28 
Reading Lower 6 43% 4 24% 19% 303 57 
Medium 5 39% 4 18% 21% 329 68 
High 2 29% 1 15% 14% 40 5 
Science Lower 5 26% 3 14% 13% 320 A 
Medium 3 25% 2 14% 11% 101 11 
High 1 10% 1 0% 10% 58 6 


Far-Off-Track Students. Differences between above- and below-average districts in the 
percentage of fourth-grade Far-Off-Track students getting On Track in eighth grade ranged 
from just under one percentage point favoring the three below-average high-poverty districts 

in mathematics to five percentage points for medium- and high-poverty districts in reading 
(Table 13).%4 In grades 8-12, differences ranged from one percentage point favoring the below- 
average medium-poverty districts in science to 12 percentage points for lower-poverty districts 


%4 Looking further into the comparison between above- and below-average high-poverty districts in mathematics, Far-Off- 
Track students in the above-average districts gained an average of 0.41 standard deviations relative to the On-Track 
target between fourth and eighth grades, compared with a gain of 0.32 standard deviations for students in the below- 
average districts (Appendix D, Table D1). Thus the apparent anomaly of more Far-Off-Track students getting On Track 
in the below-average districts is not matched in the average growth statistics, and could result from differences in the 
distribution of gains across students. 
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in reading (Table 14). In general, percentage differences were low because relatively few 
Far-Off-Track students were able to get On Track in both above- and below-average districts.°° 


For context, Tables 13 and 14 simulate how many additional Far-Off-Track students in 

the below-average districts would have gotten On Track had those districts matched the 
percentage in the above-average districts. For example, in the six below-average high-poverty 
districts in reading in grades 4-8, increasing the percentage of Far-Off-Track students getting 
On Track from 1% to 6% would have resulted in 64 additional Far-Off-Track students getting 
On Track (Table 13). 


Table 13. Percentages of Far-Off-Track Fourth-Grade Students Getting On Track 
in Grade 8, by District Poverty and Performance 
Number of Simulated 


Far-Off- Additional 
Track On-Track 


Number Number Grade 4 Grade 8 

of % of % Difference Students Students 

District Above- Getting Below- Getting in On- in Below- in Below- 

Poverty Average On Average On Track Average Average 

Subject Category Districts Track Districts Track Rates Districts Districts 
Mathematics Lower 15 6% 9 3% 3% 812 23 
Medium 16 5% 21 2% 3% 1,309 41 
High 3 3% 3 4% -1% 146 -1 
Reading Lower 6 10% 6 7% 2% 782 18 
Medium 13 9% 10 4% 5% 663 35 
High 3 6% 6 1% 5% 1,227 64 


35 Appendices D and E expand on the information shown in Tables 9-14. Appendix D provides information on prior 
and current average scores and the change in scores, while Appendix E provides information on the percentage of 
students getting On Track. Both tables provide information on the number of students in the analysis. For example, the 
denominator for 30% of Far-Off-Track students getting On Track in English in grades 8-12 in above-average lower- 
poverty districts is 23 students (first row of Tables D2 and E2). 


Table 14. Percentages of Far-Off-Track Eighth-Grade Students Getting On Track 


by Grade 12, by District Poverty and Performance 


Number of Simulated 
Far-Off- Additional 
Track On-Track 
Number Number Grade 8 Grade 12 
of % of % Difference Students Students 
District Above- Getting Below- Getting in On- in Below- _ in Below- 
Poverty Average On Average On Track Average Average 
Subject Category Districts Track Districts Track Rates Districts Districts 
English Lower 8 30% 3 NR NR 7 NR 
Medium 7 NR 8 NR NR 19 NR 
High 3 4% 4 0% 4% 21 1 
Mathematics Lower 9 7% 7 3% 5% 240 12 
Medium 12 6% 12 1% 5% 253 12 
High 5 2% 4 0% 2% 133 2 
Reading Lower 6 18% 4 5% 12% 240 29 
Medium 5 13% 4 T% T% 260 18 
High 2 11% 1 0% 11% 50 5 
Science Lower 5 12% 3 4% 8% 128 11 
Medium 3 9% 2 9% -1% 55 
High 1 6% 1 0% 6% 53 


NR = Not Reported because the calculation is based on a group of fewer than 20 students. 


Question 6: In what grades and subjects did specified percentages of 
previously On-Track students remain On Track in districts at different 


poverty and performance levels? 


In this section, we show the percentages of previously On-Track students who remained On 


Track in districts at different poverty and performance levels (Table 15), color-coding the table 
to show cases in which these percentages met or exceeded specified targets. Somewhat 
arbitrarily, we chose three target percentages: that at least 90%, 75%, or 60% of students who 
were On Track in the initial cohort grade remain On Track in the final cohort grade. By varying 
these target percentages and observing the resulting pattern of who has met the targets, 
educators and policymakers can get an idea of what kinds of targets might be realistic for the 
near future. For example, the goal that at least 90% of previously On-Track students remain 
On Track was not met in any group of districts in any subject except for English, and thus might 
be considered an aspirational goal in those other subjects. 
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Table 15. Percentages of Previously On-Track Students Staying On Track in Districts 
with Different Poverty and Performance Levels 


Lower-Poverty Medium-Poverty High-Poverty 
Grade Subject Above Average Below Above Average Below Above Average Below 
4-8 Mathematics 
Reading 
8-12 English 
Mathematics 
Reading 
Science 
NR = Not Reported because the calculation is based on a group of fewer than 20 students. Apparent discrepancies in the 
table are due to rounding: For example, below-average lower-poverty districts had slightly fewer than 60% of fourth-grade 
On-Track students in mathematics staying On Track in eighth grade. 
Criterion >90% >75% >60% | 
Table 15 can also be used to illustrate groups of districts where student performance is of 
concern because their students have not met relatively modest goals. For example, less than 
60% of fourth-grade On-Track students remained On Track in eighth grade in reading and 
mathematics in below-average medium- and high-poverty districts, and less than a third of 
eighth-grade On-Track students stayed On Track in high school mathematics in below-average 
high-poverty districts. 
Question 7: In what grades and subjects did specified percentages of 
previously Off-Track and Far-Off-Track students get On Track in districts 
at different poverty and performance levels? 
Off-Track Students. In this section, we show the percentages of previously Off-Track students 
who got On Track in districts at different poverty and performance levels (Table 16), illustrating 
cases in which these percentages met or exceeded arbitrarily selected targets that at least 
40%, 30%, or 20% of initial-grade Off-Track students get On Track in the final grade. 
Table 16. Percentages of Previously Off-Track Students Getting On Track in Districts 
with Different Poverty and Performance Levels*’ 
Lower-Poverty Medium-Poverty High-Poverty 
Grade Subject Above Average Below Above Average Below Above Average Below 
4-8 Mathematics 19 14 
Reading 13 
8-12 — English 18 9 
Mathematics 16 10 4 
Reading 29 23 15 
Science 10 12 0 
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>30% 


Criterion 240% | | 


36 These percentages are also shown for above- and below-average districts in Table 9 and 10, and for all groups of 
districts in Appendix E, Tables E7-E9. 

3’ These percentages are shown for above- and below-average districts in Table 11 and 12, and for all groups of districts 
in Appendix E, Tables E4-E6. 


Even though the students in Table 16 were not the farthest-Off-Track group—these students 
were no more than one standard deviation below the On-Track level in the initial grade—still, 
in most cases, less than 30%, and in many cases less than 20% of those students reached the 
On-Track level in the final grade. 


Far-Off-Track Students. In this section, we compare the percentages of previously Far-Off- 
Track students who got On Track in districts at different poverty and performance levels 

(Table 17), showing cases in which these percentages met or exceeded targets that at least 
15%, 10%, or 5% of students who were Off Track in the initial cohort grade get On Track in the 
final cohort grade. 


Table 17. Percentages of Previously Far-Off-Track Students Getting On Track 
in Districts with Different Poverty and Performance Levels* 


Lower-Poverty Medium-Poverty High-Poverty 


Grade Subject Above Average Below Above Average Below Above Average’ Below 


4-8 Mathematics 4 
Reading 

8-12 English 
Mathematics 


Reading 


ooo0lmoUlolUm 


Science 


NR = Not Reported because the calculation is based on a group of fewer than 20 students. 


Criterion >15% >10% >5% i 


Table 17 illustrates the low percentages of previously Far-Off-Track students who were able to 


get On Track. In only two subjects (high school English and reading) and one district category 
(above-average lower-poverty districts) did as many as 15% of these students get On Track. In 
many cases, less than 5% of Far-Off-Track students got On Track. 


Conclusion 


When performance was adjusted for differences in student demographics and prior student 
achievement, value-added performance differences between above- and below-average 
Arkansas districts were large enough to be of practical importance, often close to a year’s 
typical student growth. These differences occurred within each district poverty category, so that 
for students in high-poverty school districts, it mattered what high-poverty district they were in, 
and the same for students in lower-poverty districts. 


Likewise, differences in unadjusted student growth statistics were large enough in most cases 
to be of practical significance for students, parents, educators, and policymakers, judging 

by the number of additional On-Track students who would have stayed On Track, and the 
additional Off-Track students who would have gotten On Track, had student growth in below- 
average districts matched those in above-average districts in the same poverty category. 
Differences in percentages of Far-Off-Track students getting On Track were much smaller, 
owing to the low percentages of those students who catch up. 


38 These percentages are shown for above- and below-average districts in Table 13 and 14, and for all groups of districts 
in Appendix E, Tables E1-E3. 
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However, even in above-average, lower-poverty districts, the majority of Off-Track and Far- 
Off-Track students were unable to get On Track. In many subjects and groups of districts, less 
than 30% of Off-Track students and less than 10% of Far-Off-Track students were able to get 
On Track. 


The existence of sizeable differences in student progress across districts with similar poverty 
levels—and the need for all districts to improve—indicates the value of researching why 
students in some districts do better than students in other demographically similar districts 
(ACT, 2012a). This research might examine the improvement strategies on which district 
leaders focus and how teachers and school leaders implement those strategies. Research 
might also look at differences in strategies and implementation in high- and lower-poverty and 
rural and non-rural districts. 


In thinking about how to improve student outcomes, educators and policymakers should 
consider four basic approaches: 


1. Start early. College and career readiness does not begin in high school, or even in middle 
school (ACT, 2008). Gaps in student learning begin in early childhood and are well established 
by kindergarten (Hart & Risley, 1995; West, Denton, & Germino-Hausken, 2000). But “starting 
early” is not confined to improving early childhood and preschool programs. Improvements 
must be made in the elementary grades as well. These improvements can include 
strengthening the early reading and mathematics program, promoting better student behaviors 
and non-academic skills, and emphasizing a content-rich curriculum in the early grades that 
includes science, history/social studies, and the fine arts (Dougherty, 2013). 


2. Monitor and improve implementation of practices in key areas. These areas should 

be chosen based on their ability to improve a district's capacity to address a wide range of 

problems related to student learning. Based on research by ACT and others, these practice 
areas might include (Dougherty, 2016): 


* Developing or adopting, refining, and using a written district curriculum that describes 
what students should learn in each grade/course and subject. 

* Teaching a content-rich curriculum in the early grades. 

* Using data from multiple sources to guide improvements in teaching and learning. 


« Encouraging teachers to collaborate routinely around curriculum, instruction, and 
assessment. 


* Developing a coaching system for teachers. 


* Communicating with parents about their children’s academic progress and what their 
children are expected to learn. 


When district leaders target improvement in a given practice area, they should systematically 
gather information on what practices are actually being implemented and how that 
implementation correlates with gains in student learning (Dougherty, 2016). They should 
treat teachers and school leaders as partners in figuring out how to improve practices 
(Knight, 2007). 


3. Form networks among practitioners and researchers to share learning about 
improvement. These networks can connect educators in different districts working on the 
same problem, in addition to connecting educators in different schools in the same district. 
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The creation of cross-district knowledge-sharing networks can be particularly important in a 
state such as Arkansas that has many small geographically dispersed districts. The creation of 
such a network can be facilitated by researchers and practitioners in a state education agency, 
university, regional education laboratory, or nonprofit organization (Bryk, Gomez, Grunow, & 
LeMahieu, 2015). 


4. Work with policymakers and community leaders to strengthen out-of-school supports 
for students and their families. This approach can be particularly valuable in high-poverty 
communities, where students face out-of-school challenges that distract them from learning 
(Willingham, 2012). Strengthening support for students and their families can require better 
coordination among social service agencies and between social service agencies and schools 
(Broader, Bolder Approach to Education, 2016). 


By using these four approaches and keeping track of associations between well-implemented 
practices and improvements in student outcomes, educators and policymakers can increase 
their effectiveness in improving student learning. 
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Appendix A: Descriptive Statistics on Students 
in the Analysis 


Tables A1 and A2 show the number and percentage of students from each cohort who were 
included in the statistical analysis. Table A3 summarizes this information across cohorts to 
facilitate comparisons across grade levels. Enrollment attrition—defined as students not 
enrolled in the expected grade four years later—was higher in grades 8—12 than in 

grades 4-8. Higher enrollment attrition in high school is likely a result of students dropping 
out. In addition, fewer students took the ACT Explore in the grades 8-12 cohorts than took 
the ABE in the grades 4-8 cohorts, resulting in attrition of 28% from the grades 8—12 cohorts 
versus only 4% for the grades 4-8 cohorts (Table A3). The percentage of students taking 
ACT Explore rose substantially in 2010-11 when the state of Arkansas began paying the 
districts’ costs of giving students the ACT Explore test in eighth grade. This policy increased 
the number and percentage of students included in the 4-8 cohorts (Table A1), but not in the 
8-12 cohorts (Table A2), as students in the 8—12 cohorts were eighth graders prior to the 
2010-11 school year. 


Table A4 illustrates how student attrition affected the percentages of students in various at-risk 
groups in the statistical analysis. As would be expected, the students in the analysis—who 
were continuously enrolled in the same district, progressed by four grades in four years, and 
took all tests—were less at-risk than the general enrolled population in the initial cohort years. 
The percentages of low-income, African American, and special education students in the study 
cohorts were lower in both grades 4—8 and 8—12 than in the population from which the cohorts 
were drawn. Likewise, the percentages of Hispanic students and English language learners 
were lower in the high school study cohorts than in the population from which these cohorts 
were drawn. Sample attrition was greater in high school than in grades 4-8 for low-income, 
Hispanic, English language learner, and special education students (Table A4). 


Table A1. Percentage of Arkansas Fourth-Grade Students in the Grades 4—8 
Analysis 


Students Eligible Percent of 

Total Students Eligible for Students Students in 

Student 4th-Grade Tested in 4th Statistical in Eligible Statistical 

Cohort Enrollment and 8th Grade Analysis Districts Analysis 

2007-11 34,570 25,512 20,282 17,600 51% 
2008-12 35,418 26,499 21,133 18,340 52% 
2009-13 37,954 28,018 21,883 18,870 50% 
2010-14 37,732 26,465 21,332 18,823 50% 
Total 145,674 106,494 84,630 73,633 51% 


Notes: 


The attrition of 39,180 students between the first two data columns of this chart includes 115 students with incomplete 
demographic data; 23,706 students who were not enrolled in eighth grade four years later; 4,590 students who were 
enrolled four years later but had not taken both fourth-grade state tests; and 10,769 students enrolled and tested in grade 
4 and enrolled in grade 8, but who did not take the ACT Explore test in eighth grade. 

The attrition of 21,864 students between the second and third columns of this chart consists of students enrolled and 
tested in both grades 4 and 8 but who were not enrolled throughout grades 4-8 and tested in grades 4 and 8 in the same 
district. 
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Table A2. Percentage of Arkansas Eighth-Grade Students in Grades 8-12 Analysis 


Students Students Eligible Percent of 

Total Tested in 8th Eligible for Students Students in 

Student 8th-Grade and 11th or Statistical in Eligible Statistical 

Cohort Enrollment 12th Grade Analysis Districts Analysis 

2007-11 34,810 10,020 8,469 7,082 20% 
2008-12 35,421 10,768 9,180 8,172 23% 
2009-13 36,769 14,078 11,461 9,843 27% 
2010-14 36,882 15,328 13,248 11,280 31% 
Total 143,882 50,194 42,358 36,377 25% 


Notes: 


The attrition of 93,688 students between the first two data columns of this chart includes 155 students with incomplete 
demographic data; 37,833 students who were enrolled in eighth grade but not in twelfth grade four years later; 39,162 
students enrolled in grade 12 four years later but who had not taken ACT Explore in eighth grade; and 16,538 students 
enrolled and taking ACT Explore in grade 8 and following a normal grade progression between grades 8 and 12 but not 


taking the ACT. 


The attrition of 7,836 students between the second and third data columns of this chart consists of students who met the 
requirements for inclusion in the second column but who were not enrolled throughout grades 8-12 and tested in grades 8 


and 11 or 12 in the same district. 


Table A3. Comparing Attrition in Grades 4-8 and 8—12 Cohorts 


Grades 4-8 Grades 8-12 

Number of % of Students Number of % of Students 
Student Population Students _ in Initial Grade Students _ in Initial Grade 
All enrolled students in the initial grade 145,674 100% 143,882 100% 
...with complete demographic information 145,559 99.9% 143,727 99.9% 
...and enrolled in final grade four years later 121,853 84% 105,894 74% 
...and taking all tests in initial grade 117,263 80% 66,732 46% 
...and taking all tests in final grade 106,494 73% 50,194 35% 
...and continuously enrolled in the district 84,630 58% 42,358 29% 
...and in an eligible district 73,633 51% 36,377 25% 


Table A4. Demographics of Students in Arkansas Growth Analysis 


Grades 4-8 Grades 8-12 
Demographic Category Grade 4 Analysis Grade 8 Analysis 
% low-income 67% 60% 61% 46% 
% African American 22% 18% 22% 19% 
% Hispanic 9% 10% 8% 6% 
% Asian 2% 2% 1% 1% 
% Native American 1% 1% 1% 1% 
% English Language Learner 7% 7% 5% 2% 
% special education 14% 11% 12% 5% 


Note: The denominators for the percentages of “All students in grade 4 (or 8)” are students with complete demographic 


data. 


Appendix B: Descriptive Statistics on Districts 
in the Analysis 


Arkansas is a largely rural state whose largest district, the Little Rock School District, had 
approximately 25,000 K-12 students, averaged across the four initial cohort years.*° Overall, 
the majority of Arkansas school districts were small; only 13 (6%) of the 202 eligible districts in 
the analysis had more than 5,000 students (Figure B1). About half of the districts were in the 
medium-poverty category with 50-70% low-income students, while about one-quarter of the 
districts were in each of the lower- and high-poverty categories (Figure B2). Higher-poverty 
districts were more likely than their lower-poverty counterparts to be small and located in 
rural areas or small towns—for example, only one of the 41 high-poverty districts had more 
than 5,000 students, and 38 of the 41 high-poverty districts were located in rural areas 
(Table B1). High-poverty districts also had greater concentrations of African American and 
Hispanic students. 
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Figure B1. Distribution of eligible districts by total K-12 enrollment (N = 202 districts) 


89 District size and demographic percentages reported in this appendix are based on K-12 statistics averaged across the 
2006-07 through the 2009-10 school years. 


33 


m ACT Research Report Comparisons of Student Growth by District Performance and Poverty 


100 


64 
60 
48 
40 
Ss 32 
30 
= 15 
9 

10 : E 

_ . Eo 
‘ aa 


0-10 | >10-20 | >20-30 | >30-40 | >40-50 | >50-60 | >60-70 | >70-80 | >80-90 | > 90 


Number of Districts 
ol 
jo) 


lowest poverty lower poverty medium poverty high poverty 


Percent of Low-Income Students 


Figure B2. Distribution of eligible districts by their percentage of low-income students 
(N = 202 districts) 


Table B1. District Characteristics by District Poverty Category 


District Poverty Category 
(Percentage of Low-Income Students) 


Lower Medium High 
District characteristics (> 20-50%) (> 50%-70%) (> 70-100%) 
Number of districts 49 112 41 
eee students) ace Ween 1eBe 
% of districts > 5,000 students 10% 6% 2% 
% of districts < 1,000 students 37% 45% 61% 
% of districts rural 69% 90% 93% 
% African American students 5% 11% 40% 
% Hispanic students 3% 5% 6% 
% special education students 11% 12% 13% 


Note: Statistics are for the 202 districts eligible for the grades 4 and 8 analysis. 


Tables B2 and B3 provide district-wide demographic data aggregated across the four initial 
cohort years (the 2006-07 through the 2009-10 school years). Although three-quarters of 
districts had 50% or more low-income students, districts with substantial percentages of African 
American, Hispanic, or English language learner students were in the minority. In addition, 
Arkansas had relatively few Asian and Native American students. 
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In addition, Tables B2 and B3 provide information on the number and percentages of students 
in the analysis, aggregating across the four student cohorts at each grade level. The median 
percentages of students in the analysis shown in these tables are likely to be higher than 

the percentages of all students in the analysis shown in Tables A1 and A2, because the 
denominators for the percentages in Tables A1 and A2 include districts not in the analysis. 


Table B2. Descriptive Statistics for Arkansas School Districts in the Grades 4-8 
Analysis (N = 202) 


District Percentile 


District Statistic 5th 10th 25th 50th 75th 90th 95th 
% low-income 35% 42% 50% 58% 65% 74% 79% 
% African-American 0% 0% 1% 2% 24% 51% 67% 
% Hispanic 1% 1% 1% 2% 5% 11% 17% 
% Asian 0% 0% 0% 0% 1% 2% 3% 
% Native American 0% 0% 0% 0% 1% 1% 3% 
% English Language Learner 0% 0% 0% 1% 3% 7% 11% 
% special education 9% 9% 10% 12% 14% 15% 17% 
# students in analysis — grade 4 67 76 102 179 365 688 978 
# students in analysis — grade 8 73 84 114 196 379 754 984 


% students in analysis — grade 4 43% 47% 53% 58% 62% 64% 66% 
% students in analysis — grade 8 43% 50% 56% 61% 66% 68% 70% 


Note: The statistics shown are based on district-wide demographic data from the 2006-07 through the 2009-10 school 
years (the initial cohort years) and the number and percentages of students in the analysis for the four cohorts. In general, 
reading down the columns, the percentiles refer to different districts: For example, the district with the median percentage 
of low-income students is not necessarily the district that with the median percentage of African American students. 


Table B3. Descriptive Statistics for Arkansas School Districts in the Grades 8-12 
Analysis (N = 169) 


District Percentile 


District Statistic 5th 10th 25th 50th 75th 90th 95th 
% low-income 34% 38% 50% 59% 68% 75% 92% 
% African-American 0% 0% 1% 2% 27% 56% 75% 
% Hispanic 1% 1% 1% 2% 5% 11% 17% 
% Asian 0% 0% 0% 0% 1% 2% 3% 
% Native American 0% 0% 0% 0% 1% 2% 3% 
% English Language Learner 0% 0% 0% 1% 2% 6% 10% 
% special education 9% 10% 10% 12% 14% 15% 17% 
# students in analysis — grade 12 33 40 65 118 215 487 630 


% students in analysis — grade 12 12% 20% 29% 37% 45% 51% 54% 


Note: The statistics shown are based on district-wide demographic data from the 2006-07 through the 2009-10 school 
years (the initial cohort years) and the number and percentages of students in the analysis for the four cohorts. In general, 
reading down the columns, the percentiles refer to different districts: For example, the district with the median percentage 
of low-income students is not necessarily the district that with the median percentage of African American students. 
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Correlation of District-Level Statistics 


Tables B4 and B5 show correlations of district-level statistics used as predictors in the analysis. 
To calculate these correlations, we aggregated each statistic over the four initial cohort years, 
rather than using the yearly values of each statistic. 


Only a minority of variable pairs had at least moderately high correlations (.3 or greater in 
absolute value). Not surprisingly, districts’ percentages of English Language Learners were 
strongly related to their percentages of Hispanic students (with a correlation of .94 for the 

202 districts in the grades 4—8 analyses and the subset of 169 districts in the grades 8-12 
analysis).*° In addition, districts’ percentages of students in poverty were correlated with their 
percentages of African American students (correlation of .60 in the 202 districts and .62 in 

the subset of 169 districts), and districts with more low-income students tended to have lower 
percentages of cohort students in the analysis, as shown by the negative correlations between 
those two variables in the bottom row of Tables B4 and B5. 


Table B4. Pairwise Correlations between District-Level Statistics for Arkansas School 
Districts: Grades 4—8 Analysis (N = 202) 


% Low- % African % % % Native % % Special #in 
Income American Hispanic Asian American ELL Education Analysis 


% African American -60 
.000 
% Hispanic .04 -.09 
.528 .223 
% Asian -.16 -.20 .37 
.023 .005 .000 
% Native American -.07 -.24 16 .33 
.298 .000 .024 .000 
% English Language Learner 04 -.07 .94 49 7 
.559 .298 .000 .000 .018 
% special education .29 -.10 -.17 -.13 -.03 -.16 
.000 .160 .013 .056 .699 .025 
# students in analysis -.24 .06 42 .39 02 50 -.22 
.001 .361 .000 .000 .793 .000 .002 
% students in analysis -.52 -.49 AT 12 02 14 -.16 13 
.000 .000 .018 .095 .822 .046 .023 .072 


Notes: p-values are in italics. Correlations with p-values of .05 or less and with absolute values of .3 or higher are in bold. 


40 This might indicate the possibility of paring down the model by dropping one of those two variables. 


Table B5. Pairwise Correlations between District-Level Statistics for Arkansas School 
Districts: Grades 8—12 Analysis (Grades 8—12 Student Cohorts) (N = 169) 


% Low- % African % % % Native % % Special #in 
Income American Hispanic Asian American ELL Education Analysis 


% African American 62 
.000 
% Hispanic .05 -.08 
.516 315 
% Asian -.20 -.19 21 
.008 .012 .006 
% Native American -.06 -.25 18 39 
412 .001 .019 .000 
%English Language Learner 05 -.05 .94 .32 21 
.501 482 .000 .000 .005 
% special education 27 -.10 -.19 -.16 -.02 -.18 
.000 .207 .012 .039 157 .016 
# students in analysis -.27 .08 27 .29 .03 33 -.21 
.000 .272 .000 .000 .700 .000 .007 
% students in analysis -.30 -.15 13 12 .00 .O7 -.21 .26 
.000 .046 .100 135 .975 335 .006 .001 


Notes: p-values are in italics. Correlations with p-values of .05 or less and with absolute values of .3 or higher are in bold. 
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Appendix C: Average Scores and Changes in Scores 


The tables in this section provide information on average scores relative to On-Track targets 
and the change in average scores between grades 4 and 8 and grades 8 and 12. This 
supplements the information shown in Tables 7 and 8, providing information on starting and 
ending scores, average scores and score changes in average-performing districts, and the 
number of students involved in each comparison. 


In general, students in above-average districts made more progress relative to On-Track 
targets than did students in average districts, who in turn made more progress than did 
students in below-average districts. This was not automatically true based on how the districts 
were chosen, as it was possible for above-average districts to be demographically more 
disadvantaged than average- or below-average districts in the same poverty category, leading 
students in the above-average districts to perform worse despite the fact that they performed 
better once demographics were controlled for.*! 


In some cases, even students in above-average districts did not gain ground relative to the On- 
Track target. This occurred in grades 8-12 English at all three district poverty levels (Table C2). 


Changes in scores were more similar than were starting or ending scores for districts at 
different poverty levels in the same performance category. For example, in grades 4-8 
mathematics, the changes in scores relative to the On-Track target for above-average districts 
were 0.09, 0.10, and 0.10 in lower-, medium-, and high-poverty districts, respectively 

(Table C1). On the other hand, average fourth-grade mathematics scores for above-average 
districts in three poverty categories were -0.15, -0.34, and -0.67, indicating that students in 
above-average high-poverty districts started out over half a standard deviation below their 
lower-poverty above-average counterparts and did not significantly narrow this gap between 
grades 4 and 8. 


41 See Appendix C in Dougherty & Shaw (2016b) for an example. Appendix F in this report shows demographic data on 
the groups of districts identified in this study. 
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Table C1. Average Scores and Changes in Scores Relative to On-Track Targets 
in Grades 4-8 


Average Scores Relative to 
On-Track Target 


District District Number Number Grades 
Poverty Performance of of 4-8 

Subject Category Level Districts Students Grade 4 Grade 8 Change 
Mathematics Lower Above average 15 12,202 -0.15 -0.06 0.09 
Average 25 8,621 -0.21 -0.23 -0.01 
Below average 9 3,971 -0.30 -0.46 -0.16 
Medium Above average 16 4,743 -0.34 -0.24 0.10 
Average 75 31,035 -0.48 -0.50 -0.02 
Below average 21 4,887 -0.42 -0.63 -0.21 
High Above average 3 710 -0.67 -0.57 0.10 
Average 35 7,080 -0.75 -0.74 0.01 
Below average 3 384 -0.71 -0.81 -0.10 
Reading Lower Above average 6 3,917 -0.43 -0.08 0.34 
Average 37 17,503 -0.41 -0.18 0.23 
Below average 6 3,374 -0.39 -0.30 0.09 
Medium Above average 13 4,993 -0.59 -0.26 0.34 
Average 89 33,075 -0.69 -0.46 0.23 
Below average 10 2,597 -0.47 -0.50 -0.03 
High Above average 3 770 -0.75 -0.47 0.28 
Average 32 5,196 -0.89 -0.70 0.19 


Below average 6 2,208 -1.12 -1.06 0.05 
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Table C2. Average Scores and Changes in Scores Relative to On-Track Targets 
in Grades 8-12 English and Mathematics 


Average Scores Relative to 
On-Track Target 


District District Number Number Grades 
Poverty Performance of of 8-12 

Subject Category Level Districts Students Grade8 Grade12 Change 
English Lower Above average 8 4,932 0.81 0.77 -0.04 
Average 32 8,268 0.65 0.57 -0.09 
Below average 3 686 0.49 0.24 -0.25 
Medium Above average 7 1,134 0.54 0.48 -0.05 
Average 74 14,483 0.45 0.29 -0.15 
Below average 8 2,284 0.58 0.32 -0.26 
High Above average 3 485 -0.12 -0.16 -0.03 
Average 30 3,559 0.19 0.00 -0.19 
Below average 4 546 0.02 -0.28 -0.30 
Mathematics Lower Above average 8) 4,806 0.02 0.08 0.06 
Average 27 7,451 -0.06 -0.12 -0.06 
Below average 7 1,629 -0.21 -0.37 -0.16 
Medium Above average 12 2,683 -0.24 -0.24 -0.01 
Average 65 13,909 -0.35 -0.41 -0.06 
Below average 12 1,309 -0.42 -0.63 -0.21 
High Above average 5 413 -0.64 -0.52 0.12 
Average 28 3,676 -0.62 -0.71 -0.10 
Below average 4 501 -0.69 -0.94 -0.25 


Note: Includes eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table C3. Average Scores and Changes in Scores Relative to On-Track Targets 
in Grades 8-12 Reading and Science 


Average Scores Relative to 
On-Track Target 


District District Number Number Grades 
Poverty Performance of of 8-12 

Subject Category Level Districts Students Grade8 Grade12 Change 
Reading Lower Above average 6 3,299 -0.04 0.17 0.21 
Average 33 9,734 -0.08 0.06 0.14 
Below average 4 853 -0.32 -0.36 -0.04 
Medium Above average 5 801 -0.30 -0.10 0.20 
Average 80 16,198 -0.33 -0.23 0.10 
Below average 4 902 -0.33 -0.38 -0.05 
High Above average 2 349 -0.93 -0.63 0.30 
Average 34 4,135 -0.62 -0.58 0.04 
Below average 1 106 -0.82 -0.98 -0.16 
Science Lower Above average 5 3,466 -0.10 -0.10 -0.01 
Average 35 9,751 -0.18 -0.25 -0.07 
Below average 3 669 -0.41 -0.64 -0.24 
Medium Above average 3 1,199 -0.40 -0.41 -0.01 
Average 84 16,495 -0.44 -0.54 -0.11 
Below average 2 207 -0.58 -0.75 -0.16 
High Above average 1 184 -1.53 -0.97 0.56 
Average 35 4,281 -0.71 -0.88 -0.17 
Below average 1 125 -1.02 -1.39 -0.37 


Note: Includes eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Appendix D: Average Scores and Changes in Scores 
by Prior Achievement Level 


This appendix disaggregates the average scores and changes in scores in Appendix C by 
students’ prior achievement levels, so that the progress of students relative to On-Track targets 
can be compared for students who started out On Track, Off Track, or Far Off Track in the 
earlier grade.*? 


In nearly all cases, average score changes for students in above-average districts were 
greater than for those in average districts, which in turn were greater than those in below- 
average districts.*? This is not automatically true based on how the districts were chosen, as 
it is possible for above-average districts to be demographically more disadvantaged within 
their poverty category than average- or below-average districts. “ In addition, it is possible 
for districts that are above average with their students overall to fail to perform as well with a 
particular student prior achievement subgroup. 


In addition, Far-Off-Track students tended to make more progress on average than their Off- 
Track counterparts in the same districts, who in turn made more progress than initially On- 
Track students. For example, in the 15 above-average lower-poverty districts in mathematics 
in grades 4-8, the Far-Off-Track and Off-Track students gained 0.48 and 0.16 standard 
deviations, respectively, relative to the On-Track target between grades 4 and 8 

(Tables D1 and D4), while previously On-Track students slipped back an average of 0.12 
standard deviations relative to the target (Table D7). This pattern is consistent with the 
statistical phenomenon of regression to the mean.*° 


As was the case in the data shown in Appendix C, changes in scores were more similar 
across poverty categories than were starting or ending scores for students in the same prior 
achievement subgroup and district performance level. 


42 Thus, the average fourth- and eighth-grade scores and fourth- to eighth-grade score change of the 12,202 students in 
the 15 above-average lower-poverty school districts in mathematics (Appendix C, Table C1) are weighted averages of 
the corresponding scores and score changes of the 1,999 of those students who were Far Off Track, 4,862 who were 
Off Track, and 5,341 who were On Track in fourth grade, as shown in Tables D1, D4, and D7. 


43 The two exceptions to this pattern were Far-Off-Track students in high-poverty districts in grades 8-12 English 
(Table D2) and On-Track students in high-poverty districts in grades 8-12 reading (Table D9). 


44 Appendix F in this report shows demographic data on the groups of districts identified in this study. 


45 Because the scores of any predefined group of students contains a random component that is positive on average for 
students chosen from near the top of the score distribution and negative on average for students chosen from near 
the bottom, all averages of groups of students chosen based on their prior performance tend to move back toward the 
average of all students. Using a sports analogy, a group of baseball players chosen for the highest batting averages in 
the first six weeks of the season will probably bat at a lower average for the rest of the season, even if they continue 
to bat well above the average for all players; the reverse holds for players with the lowest batting averages in the first 
six weeks (Campbell & Kenny, 1999). This regression effect tends to reduce the average growth of students in the 
On-Track group and increase the growth of students in the Far-Off-Track group. Regression effects are enhanced when 
scores are reported in standard deviation units rather than score points. Scores measured in points can spread out due 
to Matthew effects (Stanovich, 1986) and natural increases over time in the variation in scores. Measuring scores in 
standard deviations adjusts for these widening gaps in scores, while making no adjustment to offset regression effects. 


Table D1. Average Scores and Change in Scores Relative to On-Track Targets 
in Grades 4—8: Students Who Were Far Off Track in Grade 4 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students Far Grades 
Poverty Performance of Off Track in 4-8 

Subject Category Level Districts 4th Grade Grade 4 Grade8& Change 
Mathematics Lower Above average 15 1,999 -1.53 -1.05 0.48 
Average 25 1,660 -1.55 -1.10 0.45 
Below average 9 812 -1.54 -1.20 0.34 
Medium Above average 16 1,101 -1.57 -1.04 0.53 
Average 75 8,908 -1.66 -1.28 0.38 
Below average 21 1,309 -1.63 -1.38 0.25 
High Above average 3 267 -1.65 -1.23 0.41 
Average 35 2,790 -1.71 -1.33 0.38 
Below average 3 146 -1.62 -1.30 0.32 
Reading Lower Above average 6 943 -1.68 -0.93 0.75 
Average 37 4,224 -1.58 -0.99 0.58 
Below average 6 782 -1.59 -1.04 0.55 
Medium Above average 13 1,573 -1.63 -0.95 0.68 
Average 89 11,913 -1.72 -1.14 0.58 
Below average 10 663 -1.62 -1.20 0.43 
High Above average 3 304 -1.77 -1.13 0.63 
Average 32 2,266 -1.73 -1.21 0.52 


Below average 6 1,227 -1.78 -1.40 0.38 
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Table D2. Average Scores and Change in Scores Relative to On-Track Targets in 
Grades 8-12 English and Mathematics: Students Who Were Far Off Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 
District District Number Students Far Grades 
Poverty Performance of Off Track in 8-12 
Subject Category Level Districts 8th Grade Grade 8 Grade12 Change 

English Lower Above average 8 23 -1.58 -0.51 1.08 
Average 32 71 -1.45 -0.75 0.70 

Below average 3 7 NR NR NR 

Medium Above average 7 10 NR NR NR 

Average 74 252 -1.56 -0.91 0.66 

Below average 8 19 NR NR NR 

High Above average 3 26 -1.36 -1.01 0.35 

Average 30 99 -1.55 -1.02 0.53 

Below average 4 21 -1.54 -1.27 0.27 

Mathematics Lower Above average 9 435 -1.69 -0.97 0.71 
Average 27 737 -1.68 -1.08 0.60 

Below average 7 240 -1.67 -1.14 0.52 

Medium Above average 12 399 -1.71 -1.04 0.67 

Average 65 2,685 -1.74 -1.17 0.57 

Below average 12 253 -1.79 -1.24 0.56 

High Above average 8) 114 -1.96 -1.15 0.82 

Average 28 1,036 -1.80 -1.22 0.58 

Below average 4 133 -1.91 -1.34 0.56 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
NR = Not Reported because the calculation is based on a group of fewer than 20 students. 
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Table D3. Average Scores and Change in Scores Relative to On-Track Targets in 
Grades 8-12 Reading and Science: Students Who Were Far Off Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students Off Grades 
Poverty Performance of Track in 8th 8-12 

Subject Category Level Districts Grade Grade8 Grade12 Change 
Reading Lower Above average 6 625 -1.31 -0.71 0.61 
Average 33 1,955 -1.31 -0.77 0.54 
Below average 4 240 -1.36 -1.06 0.30 
Medium Above average 8) 218 -1.31 -0.78 0.53 
Average 80 4,821 -1.34 -0.95 0.39 
Below average 4 260 -1.29 -0.98 0.31 
High Above average 2 213 -1.45 -0.91 0.54 
Average 34 1,779 -1.36 -1.08 0.27 
Below average 1 50 -1.37 -1.38 -0.01 
Science Lower Above average 5 396 -1.54 -0.93 0.61 
Average 35 1,342 -1.53 -1.01 0.51 
Below average 3 128 -1.53 -1.28 0.25 
Medium Above average 3 280 -1.61 -1.08 0.53 
Average 84 3,588 -1.59 -1.22 0.36 
Below average 2 55 -1.54 -1.24 0.30 
High Above average 1 114 -2.15 -1.13 1.02 
Average 35 1,388 -1.60 -1.39 0.21 
Below average 1 53 -1.69 -1.64 0.05 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table D4. Average Scores and Change in Scores Relative to On-Track Targets 


in Grades 4—8: Students Who Were Off Track in Grade 4 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students Off Grades 
Poverty Performance of Track in 4th 4-8 

Subject Category Level Districts Grade Grade 4 Grade 8 Change 
Mathematics Lower Above average 15 4,862 -0.45 -0.29 0.16 
Average 25 3,359 -0.45 -0.41 0.03 
Below average 9 1,675 -0.47 -0.60 -0.12 
Medium Above average 16 1,937 -0.46 -0.32 0.15 
Average 75 12,204 -0.48 -0.51 -0.03 
Below average 21 1,906 -0.48 -0.67 -0.19 
High Above average 3 269 -0.51 -0.51 0.01 
Average 35 2,697 -0.50 -0.62 -0.12 
Below average 3 151 -0.52 -0.75 -0.22 
Reading Lower Above average 6 1,631 -0.47 -0.21 0.26 
Average 37 7,342 -0.49 -0.30 0.19 
Below average 6 1,384 -0.47 -0.43 0.03 
Medium Above average 13 2,062 -0.50 -0.26 0.24 
Average 89 12,855 -0.50 -0.41 0.09 
Below average 10 1,108 -0.49 -0.59 -0.10 
High Above average 3 266 -0.51 -0.45 0.06 
Average 32 2,027 -0.54 -0.56 -0.02 
Below average 6 716 -0.54 -0.83 -0.29 


Table D5. Average Scores and Change in Scores Relative to On-Track Targets in 
Grades 8-12 English and Mathematics: Students Who Were Off Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students Off Grades 
Poverty Performance of Track in 8th 8-12 

Subject Category Level Districts Grade Grade8 Grade12 Change 
English Lower Above average 8 882 -0.49 -0.20 0.29 
Average 32 1,919 -0.50 -0.30 0.20 
Below average 3 193 -0.51 -0.52 -0.02 
Medium Above average i 307 -0.53 -0.34 0.19 
Average 74 4,451 -0.53 -0.48 0.05 
Below average 8 597 -0.51 -0.48 0.03 
High Above average 3 256 -0.61 -0.50 0.11 
Average 30 1,475 -0.56 -0.59 -0.03 
Below average 4 261 -0.57 -0.73 -0.16 
Mathematics Lower Above average 9 1,576 -0.52 -0.44 0.09 
Average 27 2,650 -0.53 -0.55 -0.02 
Below average u 628 -0.54 -0.67 -0.13 
Medium Above average 12 1,022 -0.55 -0.60 -0.05 
Average 65 5,499 -0.56 -0.70 -0.14 
Below average 12 561 -0.55 -0.82 -0.27 
High Above average 5 170 -0.56 -0.65 -0.09 
Average 28 1,532 -0.58 -0.85 -0.27 
Below average 4 225 -0.60 -1.05 -0.45 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table D6. Average Scores and Change in Scores Relative to On-Track Targets 
in Grades 8-12 Reading and Science: Students Who Were Off Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students Off Grades 
Poverty Performance of Track in 8th 8-12 

Subject Category Level Districts Grade Grade8 Grade12 Change 
Reading Lower Above average 6 1,101 -0.48 -0.16 0.32 
Average 33 3,284 -0.49 -0.24 0.25 
Below average 4 303 -0.49 -0.46 0.03 
Medium Above average 5 291 -0.50 -0.25 0.25 
Average 80 5,624 -0.50 -0.37 0.13 
Below average 4 329 -0.52 -0.55 -0.03 
High Above average 2 84 -0.54 -0.43 0.11 
Average 34 1,390 -0.53 -0.54 -0.01 
Below average 1 40 -0.60 -0.79 -0.18 
Science Lower Above average 5 1,360 -0.54 -0.49 0.05 
Average 35 Soo -0.54 -0.56 -0.01 
Below average 3 320 -0.57 -0.81 -0.24 
Medium Above average 3 496 -0.58 -0.64 -0.07 
Average 84 7,410 -0.58 -0.73 -0.15 
Below average 2 101 -0.57 -0.81 -0.24 
High Above average 1 59 -0.67 -0.84 -0.17 
Average 35 1,966 -0.61 -0.87 -0.26 
Below average 1 58 -0.66 -1.25 -0.58 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table D7. Average Scores and Change in Scores Relative to On-Track Targets 
in Grades 4—8: Students Who Were On Track in Grade 4 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students On Grades 
Poverty Performance of Track in 4th 4-8 

Subject Category Level Districts Grade Grade 4 Grade 8 Change 
Mathematics Lower Above average 15 5,341 0.63 0.51 -0.12 
Average 25 3,602 0.62 0.35 -0.27 
Below average 9 1,484 0.58 0.11 -0.48 
Medium Above average 16 1,705 0.60 0.38 -0.22 
Average 75 9,923 0.59 0.23 -0.36 
Below average 21 1,672 0.58 0.01 -0.58 
High Above average 3 174 0.60 0.35 -0.25 
Average 35 1,593 0.52 0.09 -0.43 
Below average 3 87 0.46 -0.12 -0.59 
Reading Lower Above average 6 1,343 0.50 0.67 0.17 
Average 37 5,937 0.50 0.54 0.04 
Below average 6 1,208 0.48 0.34 -0.14 
Medium Above average 13 1,358 0.48 0.56 0.08 
Average 89 8,307 0.48 0.42 -0.06 
Below average 10 826 0.49 0.18 -0.31 
High Above average 3 200 0.48 0.52 0.03 
Average 32 903 0.46 0.28 -0.19 


Below average 6 265 0.41 -0.14 -0.54 
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Table D8. Average Scores and Change in Scores Relative to On-Track Targets in 
Grades 8-12 English and Mathematics: Students Who Were On Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 

District District Number Students On Grades 
Poverty Performance of Track in 8th 8-12 

Subject Category Level Districts Grade Grade 8 Grade12 Change 
English Lower Above average 8 4,027 1.11 0.99 -0.12 
Average 32 6,278 1.03 0.85 -0.18 
Below average 3 486 0.92 0.56 -0.35 
Medium Above average if 817 0.96 0.81 -0.15 
Average 74 9,780 0.94 0.67 -0.27 
Below average 8 1,668 1.00 0.62 -0.38 
High Above average 3 203 0.65 0.39 -0.26 
Average 30 1,985 0.83 0.48 -0.35 
Below average 4 264 0.72 0.25 -0.48 
Mathematics Lower Above average 9 2,795 0.60 0.54 -0.05 
Average 27 4,064 0.54 0.33 -0.21 
Below average 7 761 0.52 0.13 -0.39 
Medium Above average 12 1,262 0.48 0.29 -0.19 
Average 65 5,725 0.50 0.23 -0.27 
Below average 12 495 0.43 -0.11 -0.54 
High Above average 8) 129 0.43 0.21 -0.23 
Average 28 1,108 0.43 -0.06 -0.49 
Below average 4 143 0.31 -0.39 -0.70 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table D9. Average Scores and Change in Scores Relative to On-Track Targets 
in Grades 8-12 Reading and Science: Students Who Were On Track in Grade 8 


Average Scores 
Relative to On-Track Target 


Number of 
District District Number Students On Grades 
Poverty Performance of Track in 8th 8-12 
Subject Category Level Districts Grade Grade8 Grade12 Change 

Reading Lower Above average 6 1,573 0.77 0.75 -0.02 
Average 33 4,495 0.74 0.64 -0.10 
Below average 4 310 0.65 0.28 -0.36 
Medium Above average 8) 292 0.67 0.57 -0.10 
Average 80 5,753 0.70 0.52 -0.18 
Below average 4 313 0.68 0.31 -0.37 
High Above average 2 52 0.56 0.21 -0.35 
Average 34 966 0.62 0.30 -0.32 

Below average 1 16 NR NR NR 
Science Lower Above average 5 1,710 (0) 2%) 0.40 -0.19 
Average 35 4,418 0.55 0.25 -0.30 

Below average 3 221 0.48 -0.03 -0.51 
Medium Above average 3 423 0.61 0.30 -0.30 
Average 84 5,497 0.50 0.14 -0.36 

Below average 2 51 0.42 -0.09 -0.51 

High Above average 1 11 NR NR NR 
Average 5) 927 0.40 -0.12 -0.52 

Below average 1 14 NR NR NR 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 


NR = Not Reported because the calculation is based on a group of fewer than 20 students. 
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Appendix E: On-Track Percentages 
by Prior Achievement Level 


The tables in this section supplement the information in Tables 9-14 by providing information 
on percentages of students getting On Track in average school districts, and on numbers of 
students in each prior achievement group. 


As expected, in most cases above-average districts had more students reaching On-Track 
status than in average and below-average districts. In cases where this pattern is reversed, in 
all but two cases the reversal is not reflected in the average growth statistics in Appendix D. In 
addition, some differences are sensitive to score changes by a small number of students. For 
example, 4% of Far-Off-Track students got On Track in grade 8 mathematics in high-poverty 
below-average districts, compared with 3% in average and above-average districts (Table E1). 
However, the order of these percentages would have changed had two fewer students in the 
below-average districts reached the On-Track level. 
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Table E1. Percentages of Far-Off-Track Fourth-Grade Students Getting On Track 
in Grade 8, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Far Getting On Getting On 
Poverty Performance of Off Track in Track in Track in 

Subject Category Level Districts 4th Grade 8th Grade 8th Grade 
Mathematics Lower Above average 15 1,999 125 6% 
Average 25 1,660 87 5% 
Below average 9 812 28 3% 
Medium Above average 16 1,101 56 5% 
Average 75 8,908 252 3% 
Below average 21 1,309 26 2% 
High Above average 3 267 9 3% 
Average 35 2,790 80 3% 
Below average 3 146 6 4% 
Reading Lower Above average 6 943 92 10% 
Average Sif 4,224 291 T% 
Below average 6 782 58 T% 
Medium Above average 13 1,573 143 9% 
Average 89 11,913 493 4% 
Below average 10 663 25 4% 
High Above average 3 304 19 6% 
Average 32 2,266 58 3% 


Below average 6 1,227 13 1% 
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Table E2. Percentages of Far-Off-Track Eighth-Grade Students Getting On Track 
in Grades 11-12 English and Mathematics, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Far Getting On Getting On 
Poverty Performance of Off Track in Track by Track by 

Subject Category Level Districts 8th Grade 12thGrade 12th Grade 
English Lower Above average 8 23 7 30% 
Average 32 71 8 11% 
Below average 3 7 NR NR 
Medium Above average 7 10 NR NR 
Average 74 252 23 9% 
Below average 8 19 NR NR 
High Above average 3 26 1 4% 
Average 30 99 6 6% 
Below average 4 21 0 0% 
Mathematics Lower Above average 9 435 32 T% 
Average 27 737 28 4% 
Below average iG 240 6 3% 
Medium Above average 12 399 23 6% 
Average 65 2,685 72 3% 
Below average 12 253 3 1% 
High Above average 5 114 2 2% 
Average 28 1,036 14 1% 
Below average 4 133 0 0% 


Notes: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
NR = Not Reported because the calculation is based on a group of fewer than 20 students. 
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Table E3. Percentages of Far-Off-Track Eighth-Grade Students Getting On Track 
in Grades 11-12 Reading and Science, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Far Getting On Getting On 
Poverty Performance of Off Track in Track by Track by 

Subject Category Level Districts 8th Grade 12thGrade 12th Grade 
Reading Lower Above average 6 625 110 18% 
Average 33 1,955 259 13% 
Below average 4 240 13 5% 
Medium Above average 5 218 29 13% 
Average 80 4,821 404 8% 
Below average 4 260 it T% 
High Above average 2 213 23 11% 
Average 34 1,779 78 4% 
Below average 1 50 0 0% 
Science Lower Above average 8) 396 49 12% 
Average 35 1,342 120 9% 
Below average 3} 128 8) 4% 
Medium Above average 3 280 24 9% 
Average 84 3,588 205 6% 
Below average 2 55 5 9% 
High Above average 1 114 a 6% 
Average 35 1,388 45 3% 
Below average 1 53 0 0% 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table E4. Percentages of Off-Track Fourth-Grade Students Getting On Track 
in Grade 8, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Off Getting On Getting On 
Poverty Performance of Track in 4th Track in Track in 

Subject Category Level Districts Grade 8th Grade 8th Grade 
Mathematics Lower Above average 15 4,862 1,868 38% 
Average 25 3,359 1,014 30% 
Below average 9 1,675 360 21% 
Medium Above average 16 1,937 685 35% 
Average 75 12,204 3,052 25% 
Below average 21 1,906 342 18% 
High Above average 3 269 54 20% 
Average 35 2,697 519 19% 
Below average 3 151 21 14% 
Reading Lower Above average 6 1,631 619 38% 
Average 37 7,342 2,489 34% 
Below average 6 1,384 394 28% 
Medium Above average 13 2,062 728 35% 
Average 89 12,855 3,657 28% 
Below average 10 1,108 225 20% 
High Above average 3 266 66 25% 
Average 32 2,027 446 22% 
Below average 6 716 96 13% 
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Table E5. Percentages of Off-Track Eighth-Grade Students Getting On Track 
in Grades 11-12 English and Mathematics, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Off Getting On Getting On 
Poverty Performance of Track in 8th Track by Track by 

Subject Category Level Districts Grade 12th Grade 12th Grade 
English Lower Above average 8 882 360 41% 
Average 32 1,919 678 35% 
Below average 3 193 45 23% 
Medium Above average 7 307 106 35% 
Average 74 4,451 1,112 25% 
Below average 8 SS) 147 25% 
High Above average 3 256 69 27% 
Average 30 1,475 268 18% 
Below average 4 261 23 9% 
Mathematics Lower Above average ¢) 1,576 492 31% 
Average 27 2,650 643 24% 
Below average i 628 120 19% 
Medium Above average 12 1,022 220 22% 
Average 65 5,499 941 17% 
Below average 12 561 62 11% 
High Above average 8) 170 28 16% 
Average 28 1,532 158 10% 
Below average 4 225 9 4% 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table E6. Percentages of Off-Track Eighth-Grade Students Getting On Track 
in Grades 11-12 Reading and Science, by District Poverty and Performance 


Number of Number Percent 
District District Number Students Off Getting On Getting On 
Poverty Performance of Track in 8th Track by Track by 

Subject Category Level Districts Grade 12th Grade 12th Grade 
Reading Lower Above average 6 1,101 470 43% 
Average 33 3,284 1,207 37% 
Below average 4 303 72 24% 
Medium Above average 5 291 113 39% 
Average 80 5,624 1,689 30% 
Below average 4 329 60 18% 
High Above average 2 84 24 29% 
Average 34 1,390 316 23% 
Below average 1 40 6 15% 
Science Lower Above average 5 1,360 360 26% 
Average 35 3,991 943 24% 
Below average 3 320 44 14% 
Medium Above average 3 496 122 25% 
Average 84 7,410 1,276 17% 
Below average 2 101 14 14% 
High Above average 1 59 6 10% 
Average 35 1,966 238 12% 
Below average 1 58 0 0% 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 
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Table E7. Percentages of On-Track Fourth-Grade Students Staying On Track 
in Grade 8, by District Poverty and Performance 


Number of Number Percent 
District District Number Students On Staying On Staying On 
Poverty Performance of Track in 4th Track in Track in 

Subject Category Level Districts Grade 8th Grade 8th Grade 
Mathematics Lower Above average 15 5,341 4,242 79% 
Average 25 3,602 2,652 74% 
Below average 9 1,484 890 60% 
Medium Above average 16 1,705 1,288 76% 
Average 75 9,923 6,691 67% 
Below average 21 1,672 934 56% 
High Above average 3 174 121 70% 
Average 35 1,593 985 62% 
Below average 3 87 51 59% 
Reading Lower Above average 6 1,343 1,040 77% 
Average 37 5,937 4,367 74% 
Below average 6 1,208 775 64% 
Medium Above average 13 1,358 1,007 74% 
Average 89 8,307 5,703 69% 
Below average 10 826 488 59% 
High Above average 3 200 143 12% 
Average 32 903 546 60% 


Below average 6 265 114 43% 


m ACT Research Report Comparisons of Student Growth by District Performance and Poverty 


60 


Table E8. Percentages of On-Track Eighth-Grade Students Staying On Track in 
Grades 11-12 English and Mathematics, by District Poverty and Performance 


Number of Number Percent 
District District Number Students On Staying On Staying On 
Poverty Performance of Track in 8th Track in Track in 

Subject Category Level Districts Grade 12th Grade 12th Grade 
English Lower Above average 8 4,027 3,646 91% 
Average 32 6,278 5,557 89% 
Below average 3 486 384 79% 
Medium Above average 7 817 694 85% 
Average 74 9,780 8,014 82% 
Below average 8 1,668 1,340 80% 
High Above average 3 203 150 74% 
Average 30 1,985 1,492 75% 
Below average 4 264 164 62% 
Mathematics Lower Above average 9 2,795 2,212 79% 
Average 27 4,064 2,937 72% 
Below average iG 761 470 62% 
Medium Above average 12 1,262 897 71% 
Average 65 5,725 3,821 67% 
Below average 12 495 249 50% 
High Above average 5 129 87 67% 
Average 28 1,108 582 53% 
Below average 4 143 46 32% 


Note: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 


Table E9. Percentages of On-Track Eighth-Grade Students Staying On Track 
in Grades 11-12 Reading and Science, by District Poverty and Performance 


Number of Number Percent 
District District Number Students On Staying On Staying On 
Poverty Performance of Track in 8th Track in Track in 

Subject Category Level Districts Grade 12th Grade 12th Grade 
Reading Lower Above average 6 1,573 1,307 83% 
Average 33 4,495 3,615 80% 
Below average 4 310 195 63% 
Medium Above average 5 292 227 718% 
Average 80 5,753 4,290 75% 
Below average 4 313 212 68% 
High Above average 2 52 34 65% 
Average 34 966 613 63% 
Below average 1 16 NR NR 
Science Lower Above average 5 1,710 1,241 73% 
Average 35 4,418 3,031 69% 
Below average 3 221 116 52% 
Medium Above average 3 423 293 69% 
Average 84 5,497 3,403 62% 
Below average 2 51 26 51% 
High Above average 1 11 NR NR 
Average 35 927 453 49% 
Below average 1 14 NR NR 


Notes: Based on eleventh-grade scores for twelfth graders who took the ACT for the last time in grade 11. 


NR = Not Reported because the calculation is based on a group of fewer than 20 students. 
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Appendix F: District-Wide Demographic Statistics 


by District Poverty and Performance 


This appendix provides district-wide demographic data across kindergarten through twelfth 


grade on the groups of districts whose value-added performance statistics and student growth 


statistics are compared in this report. These demographic statistics are averaged across the 


four initial cohort years (2006-07 through 2009-10). 


The tables show that even within poverty categories, there may be substantial differences in 


average demographics between the above-average, average, and below-average districts as 


identified by the statistical models. That is why it can be helpful to use statistical models to 


control for these demographic differences. 


Table F1. District-Wide Demographic Statistics by District Poverty and Performance 


in Grade 8 English and Mathematics 


District District English 
Poverty Performance Low- African Language —_ Special 
Subject Level Level Income American Hispanic Asian Learner [Education 
English Lower Above average 37% 7% 6% 2% 4% 11% 
Average 37% 6% 5% 1% 2% 11% 
Below average 39% 2% 2% 1% 1% 12% 
Medium Above average 57% 16% 7% 1% 4% 12% 
Average 59% 23% 12% 2% 10% 11% 
Below average 59% 16% 5% 1% 2% 12% 
High Above average 75% 38% 3% 1% 2% 13% 
Average 80% 53% 7% 1% 4% 12% 
Below average 84% 54% 11% 0% 5% 11% 
Mathematics Lower Above average 37% 8% 6% 2% 4% 11% 
Average 39% 4% 3% 1% 1% 11% 
Below average 38% 3% 3% 1% 1% 12% 
Medium Above average 58% 15% T% 1% 5% 11% 
Average 59% 23% 12% 2% 10% 11% 
Below average 60% 20% 3% 0% 1% 12% 
High Above average 85% 54% 4% 0% 3% 11% 
Average 80% 52% 7% 1% 4% 12% 
Below average 73% 31% 10% 1% 4% 15% 


Table F2. District-Wide Demographic Statistics by District Poverty and Performance 
in Grade 8 Reading and Science 


District District English 
Poverty Performance Low- African Language Special 
Subject Level Level Income American Hispanic Asian Learner Education 
Reading Lower Above average 41% 7% 6% 2% 4% 11% 
Average 37% 6% 5% 1% 2% 11% 
Below average 38% 2% 3% 1% 1% 11% 
Medium Above average 56% 18% T% 1% 4% 13% 
Average 59% 23% 12% 2% 9% 11% 
Below average 58% 9% 3% 1% 1% 12% 
High Above average 72% 40% 1% 1% 0% 11% 
Average 81% 46% 9% 1% 5% 12% 
Below average 82% 69% 5% 0% 3% 12% 
Science Lower Above average 34% 5% 6% 2% 4% 11% 
Average 40% 7% 3% 1% 1% 11% 
Below average 44% 2% 2% 1% 0% 11% 
Medium Above average 57% 19% 8% 1% 5% 11% 
Average 59% 24% 12% 2% 9% 11% 
Below average 59% 7% 3% 1% 1% 12% 
High Above average 81% 43% 3% 1% 2% 12% 
Average 78% 41% 10% 1% 5% 12% 
Below average 87% 19% 2% 0% 1% 12% 
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Table F3. District-Wide Demographic Statistics by District Poverty and Performance 
in Grades 11-12 English and Mathematics 


District District English 
Poverty Performance Low- African Language _—_ Special 
Subject Level Level Income American Hispanic Asian Learner [Education 
English Lower Above average 36% 11% 6% 2% 3% 11% 
Average 38% 3% 4% 2% 2% 11% 
Below average 38% 7% 2% 2% 0% 9% 
Medium Above average 59% 11% 3% 0% 1% 12% 
Average 60% 28% 6% 1% 4% 11% 
Below average 59% 6% 26% 2% 20% 12% 
High Above average 88% 68% 1% 0% 1% 12% 
Average 80% 51% 8% 1% 4% 12% 
Below average 81% 64% 10% 0% 5% 10% 
Mathematics Lower Above average 37% 5% 5% 2% 3% 11% 
Average 37% 7% 5% 2% 2% 11% 
Below average 35% 6% 3% 1% 2% 11% 
Medium Above average 57% 12% T% 1% 4% 11% 
Average 60% 26% 10% 1% 7% 12% 
Below average 59% 10% 5% 2% 3% 12% 
High Above average 78% 40% 5% 1% 1% 13% 
Average 81% 54% 8% 1% 5% 12% 
Below average 81% 62% 2% 0% 0% 10% 


Table F4. District-Wide Demographic Statistics by District Poverty and Performance 
in Grades 11-12 Reading and Science 


District District English 
Poverty Performance Low- African Language ___ Special 
Subject Level Level Income American Hispanic Asian Learner Education 
Reading Lower Above average 37% 6% 5% 2% 4% 12% 
Average 37% 6% 5% 2% 2% 11% 
Below average 39% T% 2% 1% 0% 9% 
Medium Above average 58% 14% 3% 1% 1% 11% 
Average 60% 24% 9% 1% 6% 12% 
Below average 62% 26% 5% 1% 3% 12% 
High Above average 76% 87% 1% 0% 0% 12% 
Average 82% 49% 9% 1% 4% 12% 
Below average 79% 62% 1% 0% 0% 11% 
Science Lower Above average 38% 13% 6% 2% 4% 12% 
Average 37% 3% 4% 1% 2% 11% 
Below average 38% 10% 2% 2% 0% 8% 
Medium Above average 63% 31% 6% 1% 3% 12% 
Average 60% 23% 9% 1% 6% 12% 
Below average 60% 0% 5% 0% 1% 15% 
High Above average 77% 97% 1% 0% 0% 12% 
Average 81% 47% 9% 1% 4% 12% 
Below average 92% 93% 1% 0% 0% 11% 
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Appendix G: Fixed-Effect Coefficients 
from Statistical Models 


Tables G1 and G2 show the fixed-effect coefficients from the hierarchical models, measured 

in units of scale score points on the test used as the dependent variable. These are partial 
effects: For example, in Table G1, the fixed-effect coefficient for “low-income status” of -0.447 
in eighth-grade English indicates that the predicted score of a low-income student is just under 
half a point (about one-ninth of a standard deviation) lower on the grade 8 ACT Explore English 
test than the predicted score of a non-low-income student who has the same values of the 
other variables in the model. No interaction effects were modeled, i.e., we did not model how a 
student’s ethnicity or prior score might affect the differences in predicted scores between low- 
and non-low-income students. 


The tables also show the standard deviation of the district random effects in each model, 
labeled as “SD of random effects (pts).” This statistic estimates the variation across districts 

in the true random effect. For example, in Table F1, the “SD of random effects (pts)” is 0.36 in 
eighth-grade English. Given the assumed normal distribution of the random effect in the model, 
for approximately two-thirds of the districts, the absolute value of the random effect is 

0.36 score points or less; for approximately 95% of the districts, the absolute value of the 
random effect is 0.72 score points or less. 


We also converted these standard deviations, measured in score points, into standardized 
form by dividing them by the standard deviation of student scores on the test in question. 
This facilitates comparisons with the sizes of the district effects of above- and below-average 
districts shown in Tables 4 and 5. For example, for eighth-grade English, the standard 
deviation of 0.36 score points translates into a standardized standard deviation of 0.09. Thus, 
the absolute value of the random effect is 0.09 of a test score standard deviation or less in 
approximately two-thirds of the districts and 0.18 standard deviations or less in approximately 
95% of the districts. 
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Table G1. Fixed-Effect Coefficients in Regressions Predicting Grade 8 Scores 


Variable English Mathematics Reading Science 
Intercept 3.444 *** 5.496 *** 6.896 *** 9.266 *** 
Low-Income Status -0.447 *** -0.428 *** -0.371 *** -0.351 *** 
African American Status -0.433 *** -0.271  *** 0.415 *** -0.264 *** 
Hispanic Status -0.140 * -0.001 -0.019 0.067 
Asian Status 0.439 = =*** 0.622 *** 0.572 *** 0.563 *** 
Native American Status -0.185 0.186 * -0.035 0.014 

ELL Status -0.194 *** -0.011 -0.090 -0.075 
Special Education Status 0.231 *** -0.389 *** 0.491 *** 0.117 *** 
Fourth-Grade Literacy Score 0.012 *** 0.004 *** 0.011 *** 0.006 *** 
Fourth-Grade Mathematics Score 0.011 *** 0.017. —*** 0.008 *** 0.011 *** 
Number Years between Tests 0.875 *** 0.963 *** 0.361 0.449 : 
District % Low-Income -0.007 *** -0.012 *** -0.008 *** -0.009 *** 
District % African American -0.006 *** -0.003 x -0.009 *** -0.003 
District % Hispanic 0.003 0.030 *** 0.011 0.008 
District % Asian 0.013 -0.009 -0.017 0.012 
District % Native American -0.022 -0.022 -0.004 0.013 
District % ELL -0.008 -0.030 ** -0.014 -0.005 
District % Special Education -0.024 *** -0.004 -0.033 *** -0.017 * 
District Avg. Grade 4 Literacy Score -0.002 *** 0.001 -0.003 *** 0.000 
District Avg. Grade 4 Mathematics Score -0.006 *** -0.009 *** -0.005 *** -0.007 *** 
District Number Students in Model 0.000 0.000 0.000 0.000 
District % Students in Model 0.001 -0.002 0.000 -0.001 

Rural District -0.351  *** -0.106 -0.154 -0.095 
Earlier Record Deleted 0.911 *** -0.887  *** -0.790 *** -0.672 *** 
Took Eighth-Grade Test in 2012 -0.376 *** -0.196 *** -0.165 *** -0.418 *** 
Took Eighth-Grade Test in 2013 -0.318 ** -0.397 _*** -0.109 * -0.217 *** 
Took Eighth-Grade Test in 2014 -0.738 =*** -0.768 *** -0.541 *** -0.499 *** 
SD of random effect (pts) 0.36 0.41 0.37 0.37 

SD of random effect (std) 0.09 0.12 0.10 0.11 


***Significant at the .01 level. **Significant at the .05 level. *Significant at the .10 level. 
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Table G2. Fixed-Effect Coefficients in Regressions Predicting Grades 11-12 Scores 


Variable English Mathematics Reading Science 
Intercept 0.265 7.817 *** 1.983 ** 5.714 *** 
Low-Income Status -0.944 *** -0.610 ** -0.382 *** -0.456 *** 
African American Status -1.367 = *** -0.761 =*** -1.298 *** -1.137 *** 
Hispanic Status -0.678 *** -0.255  *** -0.810 ** -0.631  *** 
Asian Status 0.803 *** 1.455 *** 0.398 * 0.686 *** 
Native American Status -0.406 = -0.165 -0.063 -0.312 

ELL Status -0.877 *** -0.012 -0.706 *** -0.221 
Special Education Status 1.165 *** 0.122 -0.394 *** -0.126 
Eighth-Grade English Score 0.640 *** 0.193 *** 0.396 *** 0.194 *** 
Eighth-Grade Mathematics Score 0.261 *** 0.522 *** 0.166 *** 0.361 ** 
Eighth-Grade Reading Score 0.287 =*** 0.062 *** 0.536 *** 0.190 *** 
Eighth-Grade Science Score 0.218 *** 0.298 *** 0.239 =*** 0.345 *** 
Number Years between Tests 1.135 *** 0.164 *** 0.660 *** 0.469 *** 
District % Low-Income -0.010 ** -0.021  *** -0.010 * -0.014 *** 
District % African American 0.004 0.007 = 0.002 0.005 * 
District % Hispanic -0.034 * -0.003 -0.047 = *** -0.025 * 
District % Asian 0.023 0.097 =*™ 0.033 0.073 * 
District % Native American -0.007 -0.001 0.043 0.002 
District % ELL 0.053 ** 0.026 0.081 *** 0.040 * 
District % Special Education -0.024 -0.007 0.005 -0.002 
District Avg. Grade 8 English Score -0.199 ** -0.153  *** -0.109 0.031 
District Avg. Grade 8 Mathematics Score 0.019 -0.080 0.089 -0.024 
District Avg. Grade 8 Reading Score 0.008 0.050 -0.014 -0.073 
District Avg. Grade 8 Science Score -0.095 -0.074 -0.156 . -0.118 * 
District Number Students in Model 0.000 0.000 0.000 0.001 = 
District % Students in Model -0.006 * -0.004 -0.007 _** -0.009 *** 
Rural District -0.665 *** -0.229 -0.484 *** -0.290 ** 
Earlier Record Deleted 0.479 1.015 ae 0.734 -0.113 
Twelfth Grader in 2012 -0.191 *** -0.097 * -0.048 -0.244 *** 
Twelfth Grader in 2013 -0.042 0.100 is 0.091 -0.062 
Twelfth Grader in 2014 -0.278 *** -0.168 * 0.006 -0.158 ** 
SD of random effect (pts) 0.51 0.55 0.45 0.35 

SD of random effect (std) 0.08 0.10 0.07 0.07 
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